Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How do you encrypt specific data fields in HDF (specifically data in motion) ?

avatar
 
1 ACCEPTED SOLUTION

avatar
Master Guru

HDF (NiFi) moves data around as flow files, and each flow file is made up of metadata attributes and content, where the content is just bytes. There is no internal data format where NiFi knows about "fields", so it can't provide a generic way to encrypt fields. It provides out of the box processors called EncryptContent and DecryptContent which can encrypt and decrypt the entire content of a FlowFile.

If you have a know data format like CSV, JSON, etc, and want to encrypt individual fields with in that content, it would likely require a custom processor to interpret that format and apply the encryption. It may be possible to do this with the ExecuteScript processor, but a custom Java processor would definitely be possible.

View solution in original post

5 REPLIES 5

avatar

Hi @mpandit at the moment, I believe if you're just looking to encrypt some of the information, you'd need to use something like the SplitContent processor, to split out the elements you want to encrypt, and then EncryptContent to encrypt those elements separately from the rest of the data in each flowfile. This of course means you'll potentially need to re-assemble the data when it reaches its target location.

avatar
Master Guru

Is the data encrypted when it leaves the edge device? Use SSL transport and land encrypted in HDP.

avatar

If data is encrypted and just a pass through then its ok but I am interested in explicit data encryption, like in HDP how do you apply the securoty policies using ranger.

avatar
Master Guru

HDF (NiFi) moves data around as flow files, and each flow file is made up of metadata attributes and content, where the content is just bytes. There is no internal data format where NiFi knows about "fields", so it can't provide a generic way to encrypt fields. It provides out of the box processors called EncryptContent and DecryptContent which can encrypt and decrypt the entire content of a FlowFile.

If you have a know data format like CSV, JSON, etc, and want to encrypt individual fields with in that content, it would likely require a custom processor to interpret that format and apply the encryption. It may be possible to do this with the ExecuteScript processor, but a custom Java processor would definitely be possible.

avatar

Hi,

@mpandit

You can check the following post:

https://community.hortonworks.com/questions/28500/best-practices-for-securing-data-ingestion-through...

It will provide you with more insights.

Thanks