Very nice article. If you got step by step procedure with pre-requisites,could you pls fwd to me (muthukumar.siva@gmail.com) i would like to implement in my environment. Thank you in advance. ... View more

@slachterman Above one is for AWS instances as we have been using credentials with the command. For on-prem setup I would need to check. One thing I know is when we setup the onprem servers with AWS CLI installation, we can run aws configure command to provide the credentials once and there on we can run the aws s3 commands from the command line to access AWS S3 (provided we have setup things in AWS end like IAM user creation and bucket policy etc). But with hadoop distcp the one you provided is the solution. May be we can check with AWS guys if there is an option with role based from on-prem. ... View more

@slachterman Very nice information, we have been having the same scenario and aws keys are exposed to ambari user through which we run the backup (HDFS to AWS S3) using AWS credentials. Now we have changed to Role based which means we dont need to use any credentials. Just we need to make appropriate permissions on AWS end. Just thought of sharing the info. Before "hadoop distcp -Dfs.s3a.server-side-encryption-algorithm=AES256 -Dfs.s3a.access.key=${AWS_ACCESS_KEY_ID} -Dfs.s3a.secret.key=${AWS_SECRET_ACCESS_KEY} -update hdfs://$dir/ s3a://${BUCKET_NAME}/CCS/$table_name/$year/$month/ " After " hadoop distcp -Dfs.s3a.server-side-encryption-algorithm=AES256 -update hdfs://$dir/ s3a://${BUCKET_NAME}/CCVR/$table_name/$year/$month/ " OPTIONS: <property> <name>fs.s3a.access.key</name> <description>AWS access key ID. Omit for Role-based authentication.</description> </property> <property> <name>fs.s3a.secret.key</name> <description>AWS secret key. Omit for Role-based authentication.</description> </property> ... View more