question Distcp for classified/health-care data in Archives of Support Questions (Read Only)

Distcp for classified/health-care data

smartninja723 — Tue, 25 Oct 2016 03:04:25 GMT

Hi Folks,

I have a nightly job to copy data from Cluster-1 to Cluster-2 using DistCp. Now the issue comes with secured, classified data which is stored on the Source Cluster-1 using TDE and various other techniques. Was referring to the documentation of distCp and looks like it puts the data first on the /tmp wanted to know where does it create this /tmp directory?

on Source Cluster HDFS <root>/tmp OR

<HDFS_ROOT>/<Very_secured_Data_Dir>/tmp ?

Thanks,

Re: Distcp for classified/health-care data

smartninja723 — Tue, 25 Oct 2016 03:05:36 GMT

@Simon Elliston Ball, @Jonas Straub

Re: Distcp for classified/health-care data

egarelnabi — Fri, 16 Dec 2016 15:39:32 GMT

@Smart Solutions

Not sure of the answer to that, but if you're concerned about tmp data being unencrypted/intercepted then you may consider copying it over in it's unencrypted form. This will also reduce the encryption/re-encryption overhead. The link below talks about the different options to do this.

https://community.hortonworks.com/articles/51909/how-to-copy-encrypted-data-between-two-hdp-cluster.html