question Re: Does distcp use SSL/TLS when copying files to S3 in Archives of Support Questions (Read Only)

Does distcp use SSL/TLS when copying files to S3

nsinha — Thu, 18 Feb 2016 14:45:44 GMT

I am using HW Hadoop 2.6 and use distcp to transfer files to S3. I am using the following command to transfer files:

hadoop distcp -D mapred.job.queue.name="somename" -Dfs.s3n.awsAccessKeyId="xxxx" -Dfs.s3n.awsSecretAccessKey="xxxx" -m "100" -update -pugp "hdfs:///path-in-hdfs" "s3n://path-to-s3"

Does this command transfer the data securely using https ssl/tls over port 443? Or do I need to develop a mechanism to encrypt the data before transferring?

Re: Does distcp use SSL/TLS when copying files to S3

dchaffey — Thu, 18 Feb 2016 17:48:49 GMT

Nishant, have you seen this thread on StackOverflow? It advises that distcp to S3 is secure by default.

http://stackoverflow.com/questions/28375771/securely-transferring-data-from-hdfs-to-amazon-s3-using-distcp StackOverflow user NachoNachoMan suggested you can test this by placing a policy on the bucket that prevents putobject when securetransport=false.

Also, you'll probably want to use 's3a' in your put command "s3n://path-to-s3"

https://wiki.apache.org/hadoop/AmazonS3