Alex, your question regarding the difference between option 1 and option 5 is a good one. Option 1 discusses the readers and writers that support the provenance repository implementation. These classes serialize and deserialize provenance events from Java objects to byte streams which can be written to the repository files on disk. Option 5 references the record readers and writers which are used on the NiFi canvas to support the abstract "record" concept of a collection of individual units of data within a single flowfile. In this case, the reader and writer classes convert data between external formats (JSON, CSV, arbitrary via
ScriptedRecord*) and the NiFi internal record format. I understand these concepts seem related, but these classes are completely separate and there is no overlap whatsoever. The as-yet-undeveloped EncryptedRecordReader and EncryptedRecordSetWriter classes you mention would allow you to operate on encrypted flowfile content, i.e. a flowfile contained 100 lines of customer data and some of the column values were PII/PCI/PHI and therefore encrypted. If you needed to update these records (let's say add a new property to each record which contained the last four digits of a credit card number, but the full number value was encrypted), you could use an UpdateRecord processor as follows:
EncryptedRecordReader to decrypt the records ephemerally- Add a property "lastFourDigits" which reads the
/PAN field and slices the last four digits
EncryptedRecordSetWriter would re-encrypt the sensitive fields
All of these actions happen within the lifecycle of a single
@OnTrigger of the UpdateRecord processor (even though some logic is being performed by the controller services), so none of the plaintext "record" data is persisted anywhere on the system (it is only in RAM). To be clear, in this situation, the actual implementation of the record reader and writer would need to combine the crypto capabilities with the format, so it would actually be something like EncryptedJsonRecordReader and EncryptedJsonRecordSetWriter. I haven't done the full architecture work here yet, but obviously it's not ideal to have 2*n implementations just to provide the crypto capabilities. It is likely this would require architecture changes, either to allow multiple sequentially stacked readers and writers in the processor, or the Encrypted* implementations would accept a "type-specific" record reader/writer in their definitions and perform this task via composition. That way you would maintain the type-conversion flexibility that currently exists (i.e. EncryptedRecordReader has a JsonRecordReader and the EncryptedRecordSetWriter has a CsvRecordSetWriter, etc.).
As for when NiFi persists data to disk, this is usually done during/after the
@OnTrigger phase completes in the data lifecycle. You can see code like flowFile = processSession.putAttribute(flowFile, JMS_SOURCE_DESTINATION_NAME, destinationName); or
flowFile = processSession.write(flowFile, new OutputStreamCallback() {
@Override
public void process(final OutputStream out) throws IOException {
out.write(response.getMessageBody());
}
});
This is populating the flowfile attributes or content respectively, and then on
processSession.commit() that data is persisted to whichever implementation of the content/flowfile repository that is configured (i.e. could be written to disk, volatile memory, etc.) There is a good document which goes into depth on the write-ahead log implementation, and I wrote extensively about the provenance repository serialization here. I hope this clarifies the system. Please follow up if you have further questions.
