Hi Folks, I have a nightly job to copy data from Cluster-1 to Cluster-2 using DistCp. Now the issue comes with secured, classified data which is stored on the Source Cluster-1 using TDE and various other techniques. Was referring to the documentation of distCp and looks like it puts the data first on the /tmp wanted to know where does it create this /tmp directory? on Source Cluster HDFS <root>/tmp OR <HDFS_ROOT>/<Very_secured_Data_Dir>/tmp ? Thanks, SS ... View more

Hi all, I was going through the latest documentation on Hortonworks website : http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_spark-component-guide/content/spark-encryption.html I am unable to understand the following line : - Configuring Spark for Wire Encryption "You can configure Spark to protect sensitive data in transit by enabling wire encryption. Spark supports SSL for broadcast and file server protocols, and it uses SASL encryption for the block transfer service. Note, however, that wire encryption is not yet supported for shuffle files, cached data, and other application files." - Protect sensitive data in transit by encryption ( seems to be for data ingestion part but how? From Kafka?) - park supports SSL for broadcast and file server protocols,.... (OK) -however, that wire encryption is not yet supported for shuffle files, cached data, and other application files.". So where do I get data encrypted and where data is secured and unsecured during the start of the job to execution is finished? Can someone please enlighten on this? BTW: In Spark's context, where wire encryption comes into picture? Many thanks, ... View more

Hi experts, This question is mostly related to DR and backup. We already have two clusters ( where are exactly same in configuration and one is master and another is hot standby). To mitigate the risk further, we think of a 'cold backup', where we can store the HDFS data just like previous tape based backup solutions. And want to have this stored in our data center. (not on cloud) We do not want to invest another cluster and use distcp based approach. Want to backup only hdfs data. What could be the best solution/approach/design around the same. Let me know if more inputs required. Many thanks, SS ... View more