NiFi EncryptContent Passing IV along with Secret Key

[#1]

Hello,

We are trying to use the EncryptContent processor to encrypt and decrypt PII data. We are doing Encryption in other applications using JAVA. We are trying to incorporate the same using NiFi. We are able to get EncryptContent working but not able to replicate the exact logic that we use in other applications. We need to be in sync as we share the data.

The way, we are doing it in other applications is, we use the AES-CBC-PKC5Padding Keyed encryption along with a Initialization Vector and the Secret Key used, is encrypted using Standard PBE. I would like to know how to replicate this in NiFi. How do I pass the Initialization Vector? Please help.

NOTE : I have this working as ExecuteScript and I am also able to use EncryptContent using the raw key hex value without KDF.

Thanks,

Prakash

[#2 · Accepted solution]

Hi Prakash,

I think there are multiple questions here so I've tried to split out my answers by my understanding. Please correct any erroneous assumptions I have made.

1. Currently you can prepend the IV to the cipher text as the flowfile content to provide unique IV decryption via EncryptContent (see KeyedEncryptor L131). You are correct that ExecuteScript could also provide this behavior. I have included an example unit test which does this and the resulting log output to allow you to verify that. EncryptContent in encrypt mode automatically prepends the IV to the cipher text to allow the same processor in decrypt mode to handle this seamlessly. This is followed by a static 6 byte delimiter "NiFiIV" (4E 69 46 69 49 56 in hex) which is used by the decryptor to recognize the IV (see image).

   

2. At this time, you cannot provide a static IV as a property descriptor in the EncryptContent processor. If this is desired functionality for you, you can open a Jira requesting a dynamic property to provide a static IV, but I do not believe it would be prioritized, as a static IV is not recommended best practice.
3. I am not sure what this section means:

   The way, we are doing it in other applications is, we use the AES-CBC-PKC5Padding Keyed encryption along with a Initialization Vector and the Secret Key used, is encrypted using Standard PBE. I would like to know how to replicate this in NiFi.

   If you are using the combination of IV and raw secret key, there is no PBE (Password-Based Encryption) happening. If you are using a password, then some KDF (Key Derivation Function) is being employed. If you provide both the raw key value as a property in the EncryptContent processor and select "AES-CBC" as the encryption algorithm, the KDF property must be "None".

4. If you are able to use EncryptContent with the raw key hex value and it is succeeding, then you are already providing the IV in the correct format and location.

Unit test for arbitrary IV with keyed encryption:

    @Test
    void testShouldAcceptArbitraryIVInContent() throws IOException {
        // Arrange
        final TestRunner testRunner = TestRunners.newTestRunner(new EncryptContent())
        final String RAW_KEY_HEX = "ab" * 16
        testRunner.setProperty(EncryptContent.RAW_KEY_HEX, RAW_KEY_HEX)
        testRunner.setProperty(EncryptContent.KEY_DERIVATION_FUNCTION, KeyDerivationFunction.NONE.name())
        def keyedCipherEMs = EncryptionMethod.values().findAll { it.isKeyedCipher() }
        // Act
        keyedCipherEMs.each { EncryptionMethod encryptionMethod ->
            logger.info("Attempting {}", encryptionMethod.name())
            testRunner.setProperty(EncryptContent.ENCRYPTION_ALGORITHM, encryptionMethod.name())
            testRunner.setProperty(EncryptContent.MODE, EncryptContent.ENCRYPT_MODE)
            testRunner.enqueue(Paths.get("src/test/resources/hello.txt"))
            testRunner.clearTransferState()
            testRunner.run()
            testRunner.assertAllFlowFilesTransferred(EncryptContent.REL_SUCCESS, 1)
            MockFlowFile flowFile = testRunner.getFlowFilesForRelationship(EncryptContent.REL_SUCCESS).get(0)
            logger.info("Encrypted content of flowfile: ${Hex.encodeHexString(flowFile.data)}")
            logger.info("Unique IV: ${Hex.encodeHexString(flowFile.data)[0..31]}")
            testRunner.assertQueueEmpty()
            testRunner.setProperty(EncryptContent.MODE, EncryptContent.DECRYPT_MODE)
            testRunner.enqueue(flowFile)
            testRunner.clearTransferState()
            testRunner.run()
            // Assert
            testRunner.assertAllFlowFilesTransferred(EncryptContent.REL_SUCCESS, 1)
            logger.info("Successfully decrypted {}", encryptionMethod.name())
            flowFile = testRunner.getFlowFilesForRelationship(EncryptContent.REL_SUCCESS).get(0)
            flowFile.assertContentEquals(new File("src/test/resources/hello.txt"))
        }
    }

Log output from test:

1562 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Attempting AES_CBC
2675 [pool-2-thread-1] WARN  org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - An IV was provided of length 0 bytes for encryption but should be 16 bytes
2676 [pool-2-thread-1] WARN  org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - Generating new IV. The value can be obtained in the calling code by invoking 'cipher.getIV()';
2680 [pool-2-thread-1] INFO  org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully encrypted FlowFile[0,hello.txt,38B]
2719 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Encrypted content of flowfile: af745bdb7fde42cf60dffd770eaa4f414e69466949565620ac7676b4ffee33668d0933523cf0
2724 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Unique IV: af745bdb7fde42cf60dffd770eaa4f41
2736 [pool-3-thread-1] INFO  org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully decrypted FlowFile[0,hello.txt,13B]
2737 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Successfully decrypted AES_CBC
2744 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Attempting AES_CTR
2750 [pool-4-thread-1] WARN  org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - An IV was provided of length 0 bytes for encryption but should be 16 bytes
2750 [pool-4-thread-1] WARN  org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - Generating new IV. The value can be obtained in the calling code by invoking 'cipher.getIV()';
2750 [pool-4-thread-1] INFO  org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully encrypted FlowFile[1,hello.txt,35B]
2754 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Encrypted content of flowfile: 073764cb39ab8d2879c24fa516dc5ee04e6946694956c4d4c1ce83966bd0cb2554fc0e
2754 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Unique IV: 073764cb39ab8d2879c24fa516dc5ee0
2756 [pool-5-thread-1] INFO  org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully decrypted FlowFile[1,hello.txt,13B]
2757 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Successfully decrypted AES_CTR
2757 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Attempting AES_GCM
2763 [pool-6-thread-1] WARN  org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - An IV was provided of length 0 bytes for encryption but should be 16 bytes
2763 [pool-6-thread-1] WARN  org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - Generating new IV. The value can be obtained in the calling code by invoking 'cipher.getIV()';
2765 [pool-6-thread-1] INFO  org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully encrypted FlowFile[2,hello.txt,51B]
2765 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Encrypted content of flowfile: 8c526b99b7d7534329c4a08cc345f9ba4e69466949568dab2fa15b5e940a2221d16659ca071b007fbff514009d0354515c604b
2765 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Unique IV: 8c526b99b7d7534329c4a08cc345f9ba
2767 [pool-7-thread-1] INFO  org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully decrypted FlowFile[2,hello.txt,13B]
2768 [main] INFO  org.apache.nifi.processors.standard.TestEncryptContentGroovy - Successfully decrypted AES_GCM
Process finished with exit code 0

[#3]

Hello @Andy LoPresto,

Thank you very much for the detailed answer.

1. Got it. I will try the Decrypt part and verify that.

2. I understand that, it is not the recommended practice, but yes, this is what I was looking for. I will re-evaluate and raise JIRA for requesting a dynamic property for this.

3. What I meant was, the Key is being derived/decrypted using Standard PBE and that Key is used for Encryption/Decryption.

4. Yes, I am able to use the EncryptContent successfully. It is just that, since the data is shared, I had to make sure I replicate the same logic across.

Thanks & Regards,

Prakash