javax.crypto.AEADBadTagException: Tag mismatch!

[#1]

Hi there, I need some help regarding using a flow.xml.gz from 1.13 to 1.24. First our nifi setup: 

2 node cluster running on ec2 instances in aws where we run nifi in docker containers. We have succesfully upgraded from 1.13 to 1.24 after making a backup of the flow.xml.gz file.

We then stop the nifi containers and put back the flow.xml.gz from version 1.13. Then startup nifi. Unfortunately we get the following error:

Jan 19 07:20:58 nifi-node-1 docker[27934]: 2024-01-19 07:20:58,216 INFO [main] org.eclipse.jetty.server.Server Started @78240ms

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:58,304 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow...

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:58,326 INFO [main] o.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 8888

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,036 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 10, col 20: cvc-complex-type.2.4.b: The content of element 'flowRegistry' is not complete. One of '{class}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,037 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 16, col 20: cvc-complex-type.2.4.b: The content of element 'flowRegistry' is not complete. One of '{class}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,037 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 27, col 16: cvc-complex-type.2.4.a: Invalid content was found starting with element 'value'. One of '{provided}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,037 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 33, col 16: cvc-complex-type.2.4.a: Invalid content was found starting with element 'value'. One of '{provided}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,045 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 39, col 16: cvc-complex-type.2.4.a: Invalid content was found starting with element 'value'. One of '{provided}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,045 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 45, col 19: cvc-complex-type.2.4.b: The content of element 'parameter' is not complete. One of '{provided}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,046 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 50, col 16: cvc-complex-type.2.4.a: Invalid content was found starting with element 'value'. One of '{provided}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,046 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 56, col 19: cvc-complex-type.2.4.b: The content of element 'parameter' is not complete. One of '{provided}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,046 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 61, col 16: cvc-complex-type.2.4.a: Invalid content was found starting with element 'value'. One of '{provided}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,047 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 67, col 19: cvc-complex-type.2.4.b: The content of element 'parameter' is not complete. One of '{provided}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,047 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 72, col 16: cvc-complex-type.2.4.a: Invalid content was found starting with element 'value'. One of '{provided}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,047 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 78, col 16: cvc-complex-type.2.4.a: Invalid content was found starting with element 'value'. One of '{provided}' is expected.

Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,048 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error parsing Flow Configuration at line 84, col 16: cvc-complex-type.2.4.a: Invalid content was found starting with element 'value'. One of '{provided}' is expected.

Jan 19 07:21:00 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,703 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.

Jan 19 07:21:00 nifi-node-1 docker[27934]: org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [AES/GCM/NoPadding]

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:78)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.fingerprint.FingerprintFactory.decrypt(FingerprintFactory.java:996)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.fingerprint.FingerprintFactory.getLoggableRepresentationOfSensitiveValue(FingerprintFactory.java:605)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.fingerprint.FingerprintFactory.addParameter(FingerprintFactory.java:360)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.fingerprint.FingerprintFactory.addParameterContext(FingerprintFactory.java:326)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.fingerprint.FingerprintFactory.addFlowControllerFingerprint(FingerprintFactory.java:219)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:155)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:129)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.controller.inheritance.FlowFingerprintCheck.checkInheritability(FlowFingerprintCheck.java:45)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.controller.XmlFlowSynchronizer.sync(XmlFlowSynchronizer.java:205)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:42)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1530)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:817)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:457)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:896)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.NiFi.<init>(NiFi.java:172)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.NiFi.<init>(NiFi.java:83)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.NiFi.main(NiFi.java:332)

Jan 19 07:21:00 nifi-node-1 docker[27934]: Caused by: javax.crypto.AEADBadTagException: Tag mismatch!

Jan 19 07:21:00 nifi-node-1 docker[27934]: at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(Unknown Source)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(Unknown Source)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(Unknown Source)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at java.base/com.sun.crypto.provider.CipherCore.doFinal(Unknown Source)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(Unknown Source)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at java.base/javax.crypto.Cipher.doFinal(Unknown Source)

Jan 19 07:21:00 nifi-node-1 docker[27934]: at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:74)

Jan 19 07:21:00 nifi-node-1 docker[27934]: ... 18 common frames omitted

Jan 19 07:21:00 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,733 INFO [Thread-0] org.apache.nifi.NiFi Application Server shutdown started

Any help would be much appreciated. Thank you.

Greetz

Dave

[#2 · Accepted solution]

@Dave0x1 

That is a big jump in versions from 1.13 directly to 1.24.

Use NiFi toolkit instead to change the algorithm.
https://nifi.apache.org/download/
NiFi Toolkit 1.24.0

./encrypt-config.sh -n <nifi.properties from original 1.13 NiFi> -f <flow.xml.gz from original 1.13 NiFi> -x -s <sensitive props key from NiFi> -b <bootstrap.conf from original 1.13 NiFi> -A NIFI_PBKDF2_AES_GCM_256 -g <new 1.24 flow.xml.gz filename>

Then in your NiFi 1.24 remove or rename the current flow.xml.gz and flow.json.gz files.
Place the flow.xml.gz output from above toolkit command into same location and make sure permissions and ownership are correct.

Start your NiFi 1.24.  Since the flow.json.gz does not exist, NiFi will load the flow.xml.gz and upon successful startup generate the new flow.json.gz file it will load from that point forward each time NiFi is restarted.

Hope this works for you.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Mat

[#3 · Accepted solution]

Above registry issue also solve, with the help of this thread 🤓

https://community.cloudera.com/t5/Support-Questions/NIfi-and-Nifi-Registry-Integration/m-p/286469

[#4]

After some trial and errors found where the problem lies, our nifi 1.13 uses the old alogrithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL

I tried to change this running the following command to change the flow.xml.gz and flow.json.gz to the correct algo.

# ./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256

Java home: /opt/java/openjdk
NiFi home: /opt/nifi/nifi-current

Bootstrap Config File: /opt/nifi/nifi-current/conf/bootstrap.conf

Failed to process Flow Configuration [/nifi-data/conf/flow.xml.gz]
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [AES/GCM/NoPadding]
at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:78)
at org.apache.nifi.flow.encryptor.AbstractFlowEncryptor.getOutputEncrypted(AbstractFlowEncryptor.java:31)
at org.apache.nifi.flow.encryptor.XmlFlowEncryptor.processFlow(XmlFlowEncryptor.java:57)
at org.apache.nifi.flow.encryptor.StandardFlowEncryptor.processFlow(StandardFlowEncryptor.java:50)
at org.apache.nifi.flow.encryptor.command.FlowEncryptorCommand.processFlowConfiguration(FlowEncryptorCommand.java:135)
at org.apache.nifi.flow.encryptor.command.FlowEncryptorCommand.processFlowConfigurationFiles(FlowEncryptorCommand.java:119)
at org.apache.nifi.flow.encryptor.command.FlowEncryptorCommand.run(FlowEncryptorCommand.java:96)
at org.apache.nifi.flow.encryptor.command.SetSensitivePropertiesAlgorithm.main(SetSensitivePropertiesAlgorithm.java:29)
Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(Unknown Source)
at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(Unknown Source)
at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(Unknown Source)
at java.base/com.sun.crypto.provider.CipherCore.doFinal(Unknown Source)
at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(Unknown Source)
at java.base/javax.crypto.Cipher.doFinal(Unknown Source)
at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:74)
... 7 more
NiFi Properties Processed [/opt/nifi/nifi-current/conf/nifi.properties]

So i know here it's failing, but no solution yet.

[#5 · Accepted solution]

@Dave0x1 

That is a big jump in versions from 1.13 directly to 1.24.

Use NiFi toolkit instead to change the algorithm.
https://nifi.apache.org/download/
NiFi Toolkit 1.24.0

./encrypt-config.sh -n <nifi.properties from original 1.13 NiFi> -f <flow.xml.gz from original 1.13 NiFi> -x -s <sensitive props key from NiFi> -b <bootstrap.conf from original 1.13 NiFi> -A NIFI_PBKDF2_AES_GCM_256 -g <new 1.24 flow.xml.gz filename>

Then in your NiFi 1.24 remove or rename the current flow.xml.gz and flow.json.gz files.
Place the flow.xml.gz output from above toolkit command into same location and make sure permissions and ownership are correct.

Start your NiFi 1.24.  Since the flow.json.gz does not exist, NiFi will load the flow.xml.gz and upon successful startup generate the new flow.json.gz file it will load from that point forward each time NiFi is restarted.

Hope this works for you.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Mat

[#6]

Thank you very much Mat for your advice. I'll try it out today and get back to you with the results. Sounds promising 🤓

[#7]

Hi Mat, it works! Awesome, nifi 1.24 starts up with the flow we had from 1.13 👏 Thank you very much for the advice. We are again one step further towards finalizing our migration plan.

Only thing left is is now we get the following error from the flow with regards to the nifi-registry.

Jan 22 10:57:46 awsinstancename docker[19956]: 2024-01-22 10:57:45,991 ERROR [Timer-Driven Process Thread-4] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=9311b848-f003-3b2a-ab11-af9e8884bc89,name=CheckScratchPrijs] with Flow Registry because could not retrieve version 7 of flow with identifier ab161029-c69f-4cf2-b6b3-12f444896e3e in bucket 3be9f87e-3a97-400c-a174-f335de21f23a Jan 22 10:57:46 awsinstancename docker[19956]: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) Jan 22 10:57:46 awsinstancename docker[19956]: at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)

I'll open een separate thread for this

[#8 · Accepted solution]

Above registry issue also solve, with the help of this thread 🤓

https://community.cloudera.com/t5/Support-Questions/NIfi-and-Nifi-Registry-Integration/m-p/286469

[#9 · Accepted solution]

@Dave0x1, I'm happy to see that you resolved your issue. Can you kindly mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future?