question Re: NiFi: How to put to HDFS with wire encryption? in Archives of Support Questions (Read Only)

NiFi: How to put to HDFS with wire encryption?

gkeys — Sat, 19 Nov 2016 00:40:11 GMT

In NiFi, how do I put data to HDFS with the data encrypted across the wire? NiFi cluster would be on separate cluster than HDFS.

Re: NiFi: How to put to HDFS with wire encryption?

knarayanan — Sat, 19 Nov 2016 01:05:44 GMT

I think if you setup the cluster to use wire encryption, the name node and hdfs client will handle it for you. you just need to have the updated hdfs-site.xml file available on NiFi

Re: NiFi: How to put to HDFS with wire encryption?

gkeys — Sat, 19 Nov 2016 01:29:16 GMT

I believe that is a cluster-wide setting for all client interactions with the the namenode/HDFS. I was hoping to isolate the encryption to specific flows on the NiFi side. Thoughts?

Re: NiFi: How to put to HDFS with wire encryption?

knarayanan — Sat, 19 Nov 2016 02:06:55 GMT

is this like an on prem to cloud kind of a situation. Why would you want to just encrypt specific flows? For encryption to happen, the server has to enable it. ex. https goes to 443 instead of 80. Enable hadoop wire encryption and try sending data without setting the encryption settings in the hdfs-site when you call puthdfs. I think it will keep both secure and unsecure channels open.

Re: NiFi: How to put to HDFS with wire encryption?

knarayanan — Sat, 19 Nov 2016 02:11:11 GMT

saw this in the pdfs-default xml

dfs.encrypt.data.transfer

false

Whether or not actual block data that is read/written from/to HDFS should be encrypted on the wire. This only needs to be set on the NN and DNs, clients will deduce this automatically. It is possible to override this setting per connection by specifying custom logic via dfs.trustedchannel.resolver.class.

i guess it is doable.

Re: NiFi: How to put to HDFS with wire encryption?

knarayanan — Sat, 19 Nov 2016 02:17:49 GMT

so you would have to create a class that extends TrustedChannelResolver and overrides the isTrusted() method. then set it in hdfs properties dfs.trustedchannel.resolver.class. I have not handled this before, but if you are up to it i can assist.

Re: NiFi: How to put to HDFS with wire encryption?

gkeys — Sat, 19 Nov 2016 02:45:27 GMT

I was hoping to be granular with encryption of sensitive data vs non-sensitive data flowing into HDFS for performance reasons. If performance differences are not that large ... it is no big deal, then.