HDF for teeing encrypted data?

[#1]

Hi,

How does HDF handle teeing of encrypted data from PORD to DR?

Anything special hat needs to be done in terms of key management/decrypt process?

Thanks

[#2 · Accepted solution]

@rbiswas

If you just use GetHDFS processor it should decrypt the data before moving it to DR (assuming it has the necessary read permissions), and once there, you can write it to an encryption zone. The data in-flight will be decrypted though.

Alternatively you can copy the data in it's encrypted form. You'd need to share the keys between the clusters and use HDF or DistCP to copy the files from the "/.reserved/..." folder rather than the regular folder. Take a look at the article below for clarification:

https://community.hortonworks.com/articles/51909/how-to-copy-encrypted-data-between-two-hdp-cluster....

[#3]

@Eyad Garelnabi

Thank you.

One question did you tested the GetHDFS processor for fetching from an encrypted zone?

[#4]

I haven't tested it, but I don't believe it should be an issue since the decryption should happen transparently by the platform, before the data is passed to the processor.