NiFi data encryption and PCI compliance

[#1]

How does NiFi support data protection and encryption for PCI compliance? We are looking to ingest credit card transactions data which contains sensitive and PII information.

Option 1: encrypt data at source before sending to NiFi - how does NiFi handle encrypted data for making attribute-based routing decisions?

Option 2: use NiFi to ingest raw transaction data from a secure source -> use NiFi EncryptContent processor to encrypt the content of the flow file. How does it get stored in the content and flow file repositories? Is the data encrypted when stored on disk?

[#2]

1. NiFi would have to decrypt any encrypted data before it can be pulled into an attribute for decision making. This obviously assumes that all the data is encrypted, and you are not providing a payload to NiFI combining unencrypted meta-data and the encrypted payload, which works very well.
2. You could use EncryptContent or a 3rd party protection service to encrypt the data within NiFi, this has the benefit of the action being included in the NiFi Provenance record. The Data would appear in the NiFi Content repository, so you would protect this, typically on an encrypted volume. The other repositories would not have sensitive data in them unless you copied it there, such as using ExtractText to write the unencrypted CC# to an Attribute.
3. Bonus option: Use a tool which provides format preserving encryption to allow routing decisions without exposing sensitive data, either in or before NiFi.

[#3 · Accepted solution]

Currently, NiFi supports encrypting/decrypting data through the EncryptContent processor, but the pre/post state of the data would still be stored in plaintext in the content repository. In general, transparent disk encryption/OS-level data encryption is recommended in conjunction with strict OS-level/POSIX access controls.

There is a current effort to provide encrypted implementations of the flowfile (attribute), content, and provenance repositories.

As Dan mentioned, a combination of encrypted payload and plaintext metadata for routing can work very well if the payload does not need to be processed/transformed inside NiFi.