Support Questions
Find answers, ask questions, and share your expertise

S3distcp access to S3 on multiple AWS accounts

S3distcp access to S3 on multiple AWS accounts

Explorer

Hi,

 

I am using S3DistCp on Hadoop running AWS account "A" set with AWS keys for user on another AWS account "B" to run s3n,s3a and s3, since we have to access S3 buckets on both accounts. Below you may find the content of the HDFS core_site_safety_valve. 

 

We compiled S3DistC on our cluster using code from from https://github.com/libin/s3distcp

 

CDH version is: 5.3.8-1.cdh5.3.8.p0.5

 

CM Version: Cloudera Express 5.6.0 (#54 built by jenkins on 20160211-1910 git: 1c2be84380aa23bd5d6993ec300e144c78b37bf2) 

 

Could you let us know if it is possible to add another set of AWS keys for the to HDFS core_site_safety_valve in order to avoid S3 objects ownership problems? 

 

 

Here is the content of HDFS core_site_safety_valve (keys are from another AWS account not the one Hadoop is running on):

 

<name>fs.s3n.awsAccessKeyId</name>\n <value>awsAccessKeyId</value>\n</property>\n<property>\n <name>fs.s3n.awsSecretAccessKey</name>\n<value>awsSecretAccessKey</value>\n</property>\n<property>\n

<name>fs.s3.awsAccessKeyId</name>\n <value>awsAccessKeyId</value>\n</property>\n<property>\n <name>fs.s3.awsSecretAccessKey</name>\n <value>awsSecretAccessKey</value>\n</property>\n<property>\n <name>fs.s3a.awsAccessKeyId</name>\n <value>awsAccessKeyId</value>\n</property>\n<property>\n <name>fs.s3a.awsSecretAccessKey</name>\n<value>AwsSecretAccessKey</value>\n</property>"

 

Thank you,

 

Brat

1 REPLY 1

Re: S3distcp access to S3 on multiple AWS accounts

Explorer

Actually, there is no need for adding another set keys. Property fs.s3a.acl.default can be set to define canned ACL for newly created objects.

 

More info  at https://github.com/Aloisius/hadoop-s3a/blob/master/README.md