Nifi 1.16 fails to start after changing propsKey

[#1]

Hi,

We're encountering some errors when starting Nifi 1.16.3 after calling encrypt-config.sh in a secure environment.

For background, Nifi is deployed and managed using an in-house deployment product and is used as a component within our application stack. In general, everything works fine when using Nifi 1.15.1, however, we are encountering issues when using Nifi 1.16.3.

In short, when deploying an instance of Nifi in our system, we first create all settings necessary, including a nifi.properties and bootstrap.conf files.

At this first stage, we set an initial password of nififtw! to sensitive.props.key, which at this moment in time is plain text.

We execute encrypt-config command:

JAVA_HOME=/tech/java/openjdk1.8.0_322
/bin/encrypt-config.sh --key AAAF0313BCC242CC9EC1DD30AD74FB2E --niFiProperties /servers/nifi/default/conf/nifi.properties --loginIdentityProviders /servers/nifi/default/conf/login-identity-providers.xml --bootstrapConf /servers/nifi/default/conf/bootstrap.conf

At this stage, we don't parse in the flow.xml as it doesn't exist yet, however, the protected values in nifi.properties, i.e sensitive.props.key, are encrypted based on the --key.
Nifi starts as expected.

In a later moment, when updating our system, we invoke encrypt-config.sh one more time, but now we attempt to change the propsKey to the same value of they hexkey used. Note that at this stage, we have a flow.xml and we parse in the path as an argument:

JAVA_HOME=/tech/java/openjdk1.8.0_322
/bin/encrypt-config.sh --key AAAF0313BCC242CC9EC1DD30AD74FB2E --niFiProperties /servers/nifi/default/conf/nifi.properties --loginIdentityProviders /servers/nifi/default/conf/login-identity-providers.xml --flowXml /databases/nifi/default/flow.xml.gz --propsKey AAAF0313BCC242CC9EC1DD30AD74FB2E --bootstrapConf /servers/nifi/default/conf/bootstrap.conf

No errors are encountered at this stage and the value of sensitive.props.key in nifi.properties is changed accordingly. However, when attempting to start nifi server, we encounter the error bellow:
nifi-bootstrap.log

2022-11-13 06:32:06,488 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 82873
2022-11-13 06:32:14,269 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 32908
2022-11-13 06:34:06,217 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
2022-11-13 06:34:06,218 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down...
2022-11-13 06:34:06,945 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi

nifi-app.log

2022-11-13 06:34:06,209 INFO [main] org.eclipse.jetty.server.Server Started @119858ms
2022-11-13 06:34:06,209 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:78)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.decrypt(StandardFlowComparator.java:281)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareProperties$3(StandardFlowComparator.java:291)
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareProperties(StandardFlowComparator.java:289)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:267)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$1(StandardFlowComparator.java:114)
at java.util.HashMap.forEach(HashMap.java:1290)

at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:467)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compare$5(StandardFlowComparator.java:472)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$1(StandardFlowComparator.java:114)
at java.util.HashMap.forEach(HashMap.java:1290)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareComponents(StandardFlowComparator.java:112)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:472)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:94)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:79)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.compareFlows(VersionedFlowSynchronizer.java:387)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:167)
at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1524)
at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:107)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:819)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:542)
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008)
at org.apache.nifi.NiFi.<init>(NiFi.java:170)
at org.apache.nifi.NiFi.<init>(NiFi.java:82)
at org.apache.nifi.NiFi.main(NiFi.java:330)
Caused by: javax.crypto.BadPaddingException: pad block corrupted
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.doFinal(Unknown Source)
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2168)
at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:74)
... 62 common frames omitted

As shown in the errors above, the decryption fails when attempting to start the nifi server.

We suspect that when we try to encrypt the flow.xml with a new sensitive.props.key, the re-encryption doesn't happen, even though the value of propsKey in nifi.properties has been replaced and encrypted accordingly to the new password we parse.

After we get the failure described, if we manually change the propsKey in nifi.properties to what it was before (the encrypted version of 'nififtw!'), we manage to start nifi successfully.
Note that the password nififtw! is just one example. The issue happens with any password we might use for the first call of encrypt-config, even if we call encrypt-config again specifying a new propsKey.

We've attempted to modify the arguments used when executing the encrypt-config.sh, i.e. by creating the modified flow.xml and nifi.properties in a /tmp/ directory and manually replacing the older versions, which hasn't worked either.

This issue only arrises in Nifi version 1.16.3.0. We were using version 1.15.1.0 and version 1.11.3.0 and the same logic works perfectly fine.

This is how my nifi.properties looks like:

# security properties #
nifi.sensitive.props.key=<encrypted-key>
nifi.sensitive.props.key.protected=aes/gcm/128
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.additional.keys=

Any idea of what might be causing this issue?