How do you encrypt specific data fields in HDF (specifically data in motion) ?

[#1]

Hi @mpandit at the moment, I believe if you're just looking to encrypt some of the information, you'd need to use something like the SplitContent processor, to split out the elements you want to encrypt, and then EncryptContent to encrypt those elements separately from the rest of the data in each flowfile. This of course means you'll potentially need to re-assemble the data when it reaches its target location.

[#2]

Is the data encrypted when it leaves the edge device? Use SSL transport and land encrypted in HDP.

[#3]

If data is encrypted and just a pass through then its ok but I am interested in explicit data encryption, like in HDP how do you apply the securoty policies using ranger.

[#4 · Accepted solution]

HDF (NiFi) moves data around as flow files, and each flow file is made up of metadata attributes and content, where the content is just bytes. There is no internal data format where NiFi knows about "fields", so it can't provide a generic way to encrypt fields. It provides out of the box processors called EncryptContent and DecryptContent which can encrypt and decrypt the entire content of a FlowFile.

If you have a know data format like CSV, JSON, etc, and want to encrypt individual fields with in that content, it would likely require a custom processor to interpret that format and apply the encryption. It may be possible to do this with the ExecuteScript processor, but a custom Java processor would definitely be possible.

[#5]

Hi,

@mpandit

You can check the following post:

https://community.hortonworks.com/questions/28500/best-practices-for-securing-data-ingestion-through...

It will provide you with more insights.

Thanks