Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop to AWS S3 access denied

Highlighted

Sqoop to AWS S3 access denied

New Contributor

Hi Team,
I tried running below sqoop command (redered Cloudera decumentation https://www.cloudera.com/documentation/enterprise/latest/topics/admin_sqoop_s3_import.html)

sqoop import -Dfs.s3a.access.key=<access key> -Dfs.s3a.secret.key=<secret key> --connect jdbc:mysql://<Mysql IP>:3306/<MySQL DB> --username <UserName> --password <password> --table <MySQL Table> --target-dir s3a://<my S3 Bucket>/widget12345 --split-by id

it failed with below error

19/07/30 16:04:04 ERROR tool.ImportTool: Import failed: java.nio.file.AccessDeniedException: s3a://AKIAUKSCAK3BPICCE2UR:7HGxejD+M98V+WDa1fA9Yzn8UJFtCFnXiLFSnk4j@manoj1909/hello: getFileStatus on s3a://AKIAUKSCAK3BPICCE2UR:7HGxejD+M98V+WDa1fA9Yzn8UJFtCFnXiLFSnk4j@manoj1909/hello: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 41AFBD35A5405B5F; S3 Extended Request ID: 6hWmnB6wPsaRjIkK8P6EgO4wxO+UCAlnwFY7tYbcyeTYwxhO2PAZMtm39Tgr7vTmQE9wvJU5oMs=), S3 Extended Request ID: 6hWmnB6wPsaRjIkK8P6EgO4wxO+UCAlnwFY7tYbcyeTYwxhO2PAZMtm39Tgr7vTmQE9wvJU5oMs=:403 Forbidden

        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:218)

        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:145)

        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2184)

        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149)

        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)

 

I tried trouble shooting using this referece https://hadoop.apache.org/docs/r3.1.2/hadoop-aws/tools/hadoop-aws/troubleshooting_s3a.html but couldnt find a resolution.

 

hadoop distcp /user/<UserName>/path-to-file s3a://s3_aws_access_key_id:s3_aws_access_key_secret@my_bucketname/some-directory

distcp proves that S3 bucket has write access and IAM user is setup with necessary permission but Sqoop job with same access key and secret key fails. 

 

Note: I do not have admin access to cluster where sqoop is installed thus I cannot edit core-site.xml to have below preperties

<property>
    <name>fs.s3a.access.key</name>
    <value> <access key> </value>
</property>
<property>
    <name>fs.s3a.secret.key</name>
    <value> <secret-key> </value>
</property>

Please let me know if  am missing something during sqoop import or suggest alternatives.