Nifi 1.16 fails to start with Decryption exception

[#1]

We are encountering some issues when starting Nifi 1.16.3 after calling encrypt-config.sh

For background, Nifi is deployed and managed using an in-house deployment product, and is used as a component within our application stack. In general everything works fine when using Nifi 1.15.1, however, we are encountering issues when using Nifi 1.16.3. 

See the following in nifi-bootstrap.log

2022-11-13 06:32:06,488 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 82873
2022-11-13 06:32:14,269 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 32908
2022-11-13 06:34:06,217 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
2022-11-13 06:34:06,218 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down...
2022-11-13 06:34:06,945 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi

And the following in nifi-app.log

2022-11-13 06:34:06,209 INFO [main] org.eclipse.jetty.server.Server Started @119858ms
2022-11-13 06:34:06,209 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
        at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:78)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.decrypt(StandardFlowComparator.java:281)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareProperties$3(StandardFlowComparator.java:291)
        at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareProperties(StandardFlowComparator.java:289)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:267)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$1(StandardFlowComparator.java:114)
        at java.util.HashMap.forEach(HashMap.java:1290)

        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:467)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compare$5(StandardFlowComparator.java:472)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$1(StandardFlowComparator.java:114)
        at java.util.HashMap.forEach(HashMap.java:1290)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareComponents(StandardFlowComparator.java:112)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:472)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:94)
        at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:79)
        at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.compareFlows(VersionedFlowSynchronizer.java:387)
        at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:167)
        at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
        at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1524)
        at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:107)
        at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:819)
        at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:542)
        at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
        at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:423)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.Server.doStart(Server.java:387)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008)
        at org.apache.nifi.NiFi.<init>(NiFi.java:170)
        at org.apache.nifi.NiFi.<init>(NiFi.java:82)
        at org.apache.nifi.NiFi.main(NiFi.java:330)
Caused by: javax.crypto.BadPaddingException: pad block corrupted
        at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.doFinal(Unknown Source)
        at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
        at javax.crypto.Cipher.doFinal(Cipher.java:2168)
        at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:74)
        ... 62 common frames omitted

We suspect the issue is caused by making multiple calls encrypt-config.sh during the application lifecycle. 

JAVA_HOME=/tech/java/openjdk1.8.0_322 /dr01/qadapps/systest/build/catalog/packages/nifi-toolkit/1/16/3/0/bin/encrypt-config.sh --verbose --key 3AC8237A33D0405081562FDA4744DCF9 --niFiProperties /dr01/qadapps/systest/servers/nifi/default/conf/nifi.properties --loginIdentityProviders /dr01/qadapps/systest/servers/nifi/default/conf/login-identity-providers.xml --bootstrapConf /dr01/qadapps/systest/servers/nifi/default/conf/bootstrap.conf

...

JAVA_HOME=/tech/java/openjdk1.8.0_322 /dr01/qadapps/systest/build/catalog/packages/nifi-toolkit/1/16/3/0/bin/encrypt-config.sh --verbose --key 3AC8237A33D0405081562FDA4744DCF9 --niFiProperties /dr01/qadapps/systest/servers/nifi/default/conf/nifi.properties --loginIdentityProviders /dr01/qadapps/systest/servers/nifi/default/conf/login-identity-providers.xml --flowXml /dr01/qadapps/systest/databases/nifi/default/flow.xml.gz --propsKey 3AC8237A33D0405081562FDA4744DCF9 --bootstrapConf /dr01/qadapps/systest/servers/nifi/default/conf/bootstrap.conf

Are there any issues calling encrypt-config.sh multiple times? Or any issues setting the "nifi.sensitive.props.key" to "nififtw!"?

[#2 · Accepted solution]

Hi @mmoura , The error that you encountered in nifi 1.16 is appearing because of some changes introduced in nifi starting from 1.16 version.

Starting from nifi 1.16 , NIFI start writing on flow.xml.gz as well as JSON format ( prior to 1.16 this JSON was not there ) flow.json.gz , see the link https://www.mail-archive.com/users@nifi.apache.org/msg15332.html

When 1.16 nifi start it creates both the file, writes onto it and then encrypt it using props key from nifi.properties , when your code run encrypt-config.sh tool it only changes the flow.xml.gz with new props key and JSON flow file still encrypted with old props key, While next time nifi going to start it try to read props key from nifi.properties , which is the new ( changed props key ) and it successfully able to decrypt the flow.xml.gz but it can not able to decrypt flow.json.gz because JSON file is still encrypted with old props key which is overwritten by new props key on nifi.properties

Solution : from 1.16 onwards "./bin/nifi.sh set-sensitive-properties-key NewSensitivePropertiesKey" tool improved to cover both xml as well as json version of flow file while changing the props key, see JIRA ticket https://issues.apache.org/jira/browse/NIFI-9711 , So while changing the props key you can try using nifi.sh in place of encrypt-config.sh

Regards

Vikas