Member since
07-30-2019
155
Posts
107
Kudos Received
33
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 7459 | 04-18-2019 08:27 PM | |
| 2491 | 12-31-2018 07:36 PM | |
| 4332 | 12-03-2018 06:47 PM | |
| 1399 | 06-02-2018 02:35 AM | |
| 3593 | 06-02-2018 01:30 AM |
10-26-2017
12:40 AM
Alex, your question regarding the difference between option 1 and option 5 is a good one. Option 1 discusses the readers and writers that support the provenance repository implementation. These classes serialize and deserialize provenance events from Java objects to byte streams which can be written to the repository files on disk. Option 5 references the record readers and writers which are used on the NiFi canvas to support the abstract "record" concept of a collection of individual units of data within a single flowfile. In this case, the reader and writer classes convert data between external formats (JSON, CSV, arbitrary via
ScriptedRecord* ) and the NiFi internal record format. I understand these concepts seem related, but these classes are completely separate and there is no overlap whatsoever. The as-yet-undeveloped EncryptedRecordReader and EncryptedRecordSetWriter classes you mention would allow you to operate on encrypted flowfile content, i.e. a flowfile contained 100 lines of customer data and some of the column values were PII/PCI/PHI and therefore encrypted. If you needed to update these records (let's say add a new property to each record which contained the last four digits of a credit card number, but the full number value was encrypted), you could use an UpdateRecord processor as follows:
EncryptedRecordReader to decrypt the records ephemerally
Add a property "lastFourDigits" which reads the /PAN field and slices the last four digits
EncryptedRecordSetWriter would re-encrypt the sensitive fields
All of these actions happen within the lifecycle of a single
@OnTrigger of the UpdateRecord processor (even though some logic is being performed by the controller services), so none of the plaintext "record" data is persisted anywhere on the system (it is only in RAM). To be clear, in this situation, the actual implementation of the record reader and writer would need to combine the crypto capabilities with the format, so it would actually be something like EncryptedJsonRecordReader and EncryptedJsonRecordSetWriter . I haven't done the full architecture work here yet, but obviously it's not ideal to have 2*n implementations just to provide the crypto capabilities. It is likely this would require architecture changes, either to allow multiple sequentially stacked readers and writers in the processor, or the Encrypted* implementations would accept a "type-specific" record reader/writer in their definitions and perform this task via composition. That way you would maintain the type-conversion flexibility that currently exists (i.e. EncryptedRecordReader has a JsonRecordReader and the EncryptedRecordSetWriter has a CsvRecordSetWriter , etc.).
As for when NiFi persists data to disk, this is usually done during/after the
@OnTrigger phase completes in the data lifecycle. You can see code like flowFile = processSession.putAttribute(flowFile, JMS_SOURCE_DESTINATION_NAME, destinationName); or
flowFile = processSession.write(flowFile, new OutputStreamCallback() { @Override public void process(final OutputStream out) throws IOException { out.write(response.getMessageBody()); } });
This is populating the flowfile attributes or content respectively, and then on
processSession.commit() that data is persisted to whichever implementation of the content/flowfile repository that is configured (i.e. could be written to disk, volatile memory, etc.) There is a good document which goes into depth on the write-ahead log implementation, and I wrote extensively about the provenance repository serialization here. I hope this clarifies the system. Please follow up if you have further questions.
... View more
10-21-2017
07:05 PM
The ExecuteScript processor does not support PHP. However, you can use ExecuteProcess or ExecuteStreamCommand to run the php command line application to execute a script.
... View more
10-11-2017
09:54 PM
Laurent, feel free to contact me directly at alopresto@apache.org for further discussion.
... View more
10-11-2017
04:57 PM
1 Kudo
Hi Laurent, Apache NiFi 1.4.0 uses Jetty 9.4.2 to provide the underlying web server, and Jetty after versions 9.4.0 only supports TLS v1.2 for incoming connections. I would recommend using a proxy with TLS termination which accepts incoming TLS v1.1 connections and re-establishing a connection to your NiFi service which uses TLS v1.2. * Ticket - NIFI-3361 Upgrade Jetty https://issues.apache.org/jira/browse/NIFI-3361 * Ticket - NIFI-3720 Update documentation for TLS protocol version changes https://issues.apache.org/jira/browse/NIFI-3720 * Jetty Documentation TLS and SSL Versions https://www.eclipse.org/jetty/documentation/current/configuring-ssl.html#tls-and-ssl-versions * Apache NiFi Release Notes for 1.2.0 noting TLS protocol version changes https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.2.0 * Apache NiFi Migration Guidance for 1.1.0 -> 1.2.0 noting the change https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance The actual announcement that Jetty changed the protocol versions supported is buried in their release notes somewhere.
... View more
10-03-2017
06:50 PM
1 Kudo
Hi Prakash,
I think there are multiple questions here so I've tried to split out my answers by my understanding. Please correct any erroneous assumptions I have made.
Currently you can prepend the IV to the cipher text as the flowfile content to provide unique IV decryption via EncryptContent (see KeyedEncryptor L131). You are correct that ExecuteScript could also provide this behavior. I have included an example unit test which does this and the resulting log output to allow you to verify that. EncryptContent in encrypt mode automatically prepends the IV to the cipher text to allow the same processor in decrypt mode to handle this seamlessly. This is followed by a static 6 byte delimiter "NiFiIV" ( 4E 69 46 69 49 56 in hex) which is used by the decryptor to recognize the IV (see image).
At this time, you cannot provide a static IV as a property descriptor in the EncryptContent processor. If this is desired functionality for you, you can open a Jira requesting a dynamic property to provide a static IV, but I do not believe it would be prioritized, as a static IV is not recommended best practice.
I am not sure what this section means:
The way, we are doing it in other applications is, we use the AES-CBC-PKC5Padding Keyed encryption along with a Initialization Vector and the Secret Key used, is encrypted using Standard PBE. I would like to know how to replicate this in NiFi.
If you are using the combination of IV and raw secret key, there is no
PBE (Password-Based Encryption) happening. If you are using a password, then some KDF (Key Derivation Function) is being employed. If you provide both the raw key value as a property in the EncryptContent processor and select " AES-CBC " as the encryption algorithm, the KDF property must be " None ".
If you are able to use EncryptContent with the raw key hex value and it is succeeding, then you are already providing the IV in the correct format and location.
Unit test for arbitrary IV with keyed encryption:
@Test
void testShouldAcceptArbitraryIVInContent() throws IOException {
// Arrange
final TestRunner testRunner = TestRunners.newTestRunner(new EncryptContent())
final String RAW_KEY_HEX = "ab" * 16
testRunner.setProperty(EncryptContent.RAW_KEY_HEX, RAW_KEY_HEX)
testRunner.setProperty(EncryptContent.KEY_DERIVATION_FUNCTION, KeyDerivationFunction.NONE.name())
def keyedCipherEMs = EncryptionMethod.values().findAll { it.isKeyedCipher() }
// Act
keyedCipherEMs.each { EncryptionMethod encryptionMethod ->
logger.info("Attempting {}", encryptionMethod.name())
testRunner.setProperty(EncryptContent.ENCRYPTION_ALGORITHM, encryptionMethod.name())
testRunner.setProperty(EncryptContent.MODE, EncryptContent.ENCRYPT_MODE)
testRunner.enqueue(Paths.get("src/test/resources/hello.txt"))
testRunner.clearTransferState()
testRunner.run()
testRunner.assertAllFlowFilesTransferred(EncryptContent.REL_SUCCESS, 1)
MockFlowFile flowFile = testRunner.getFlowFilesForRelationship(EncryptContent.REL_SUCCESS).get(0)
logger.info("Encrypted content of flowfile: ${Hex.encodeHexString(flowFile.data)}")
logger.info("Unique IV: ${Hex.encodeHexString(flowFile.data)[0..31]}")
testRunner.assertQueueEmpty()
testRunner.setProperty(EncryptContent.MODE, EncryptContent.DECRYPT_MODE)
testRunner.enqueue(flowFile)
testRunner.clearTransferState()
testRunner.run()
// Assert
testRunner.assertAllFlowFilesTransferred(EncryptContent.REL_SUCCESS, 1)
logger.info("Successfully decrypted {}", encryptionMethod.name())
flowFile = testRunner.getFlowFilesForRelationship(EncryptContent.REL_SUCCESS).get(0)
flowFile.assertContentEquals(new File("src/test/resources/hello.txt"))
}
}
Log output from test:
1562 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Attempting AES_CBC
2675 [pool-2-thread-1] WARN org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - An IV was provided of length 0 bytes for encryption but should be 16 bytes
2676 [pool-2-thread-1] WARN org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - Generating new IV. The value can be obtained in the calling code by invoking 'cipher.getIV()';
2680 [pool-2-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully encrypted FlowFile[0,hello.txt,38B]
2719 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Encrypted content of flowfile: af745bdb7fde42cf60dffd770eaa4f414e69466949565620ac7676b4ffee33668d0933523cf0
2724 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Unique IV: af745bdb7fde42cf60dffd770eaa4f41
2736 [pool-3-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully decrypted FlowFile[0,hello.txt,13B]
2737 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Successfully decrypted AES_CBC
2744 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Attempting AES_CTR
2750 [pool-4-thread-1] WARN org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - An IV was provided of length 0 bytes for encryption but should be 16 bytes
2750 [pool-4-thread-1] WARN org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - Generating new IV. The value can be obtained in the calling code by invoking 'cipher.getIV()';
2750 [pool-4-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully encrypted FlowFile[1,hello.txt,35B]
2754 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Encrypted content of flowfile: 073764cb39ab8d2879c24fa516dc5ee04e6946694956c4d4c1ce83966bd0cb2554fc0e
2754 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Unique IV: 073764cb39ab8d2879c24fa516dc5ee0
2756 [pool-5-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully decrypted FlowFile[1,hello.txt,13B]
2757 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Successfully decrypted AES_CTR
2757 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Attempting AES_GCM
2763 [pool-6-thread-1] WARN org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - An IV was provided of length 0 bytes for encryption but should be 16 bytes
2763 [pool-6-thread-1] WARN org.apache.nifi.security.util.crypto.AESKeyedCipherProvider - Generating new IV. The value can be obtained in the calling code by invoking 'cipher.getIV()';
2765 [pool-6-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully encrypted FlowFile[2,hello.txt,51B]
2765 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Encrypted content of flowfile: 8c526b99b7d7534329c4a08cc345f9ba4e69466949568dab2fa15b5e940a2221d16659ca071b007fbff514009d0354515c604b
2765 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Unique IV: 8c526b99b7d7534329c4a08cc345f9ba
2767 [pool-7-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=ba25adf1-2e42-41f8-a14f-c9b247360829] successfully decrypted FlowFile[2,hello.txt,13B]
2768 [main] INFO org.apache.nifi.processors.standard.TestEncryptContentGroovy - Successfully decrypted AES_GCM
Process finished with exit code 0
... View more
08-27-2017
07:56 PM
Use a ReplaceText processor to replace the content of the incoming flowfile with the static response body. Because you can provide multiple outgoing connections from the previous processor, you will not lose the flowfile content, but this is the way to control the response via HandleHTTPResponse .
... View more
08-25-2017
06:40 PM
1 Kudo
The flowfile content should be returned as the response body. Have you provided content on the incoming flowfile?
... View more
08-25-2017
06:38 PM
Using your browser's Developer Tools window, use the UI to clear a queue while monitoring the network tab. Everything the Apache NiFi UI does is performed via the REST API. You will be able to see exactly what requests are sent to the server to clear the connection queue and can recreate that programmatically. The specific API endpoint you want in this case is POST /flowfile-queues/{id}/drop-requests where {id} is the connection ID.
... View more
08-23-2017
07:51 PM
D H, You bring up a good point that the EKU restriction should probably be made more evident in the docs. It makes sense -- the certificates are used in both server and client identification roles, so the roles must both be present if either is populated, but many people are unfamiliar with the EKU behavior (or unaware that the certificates serve dual purposes). I hope you are able to resolve your issue quickly.
... View more
08-23-2017
07:45 PM
1 Kudo
From
my answer to this question on Stack Overflow:
To extract the desired value, use the XPath expression
//ErrorCode . This will return a String value -7. By selecting Destination flowfile-attribute, you can keep the flowfile content constant and put this new value in a flowfile attribute (i.e. named attribute ).
You can chain the matched relationship to an UpdateAttribute processor which has the expression ${attribute:toNumber()} to convert it to a numerical representation, i.e. ${attribute:toNumber():plus(10)} would return 3.
... View more