Reply
Explorer
Posts: 6
Registered: ‎11-21-2015

Unable to copy data from HDFS to S3

I am trying to copy data from hdfs to s3 While using the distcp command, the command works for individual files.

 

So, hadoop distcp /user/username/file.txt s3a://xxxxx works fine

 

But when I try to copy the entire director structure it fails to create the directory giving the error: Error: java.io.IOException: mkdir failed for s3a://bucket****/ Error Code: 403 Forbidden; Request ID: 447400E9C5995ED9), S3 Extended Request ID: T0hsw+XaBMrkMUhDcJBKGIRtSF58dKedZdCH2qC32v9uVkwR94SGiI7Xxe8lqaFaDyjwS3oCpkg=

 

Even if I do a simple mkdir command in s3 it fails giving the same 403 forbidden issue.

So not sure what is the root cause.

 

I am able to copy the files but not able to create directories.

Cloudera Employee
Posts: 5
Registered: ‎09-09-2015

Re: Unable to copy data from HDFS to S3

You may not have all the permissions you need on the bucket.  Make sure you have "s3:ListBucket" on resource "arn:aws:s3:::<bucket-name>", as well as "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:PutObjectAcl" on resource "arn:aws:s3:::<bucket-name>/*"

 

You probably also want to add the permissions for creating and deleting multipart uploads. I'd add priviliges for the operations mentioned in the multipart upload docs as well.

New Contributor
Posts: 1
Registered: ‎02-04-2019

Re: Unable to copy data from HDFS to S3

How did you copy hdfs files to s3 ? using hadoop Distcp ?

can u please tell me what configuration did you do  ? other than passing access and secret key in core-site.xml 

Also can u please tell me if you know ? how to sync directories from hdfs to s3 like below :

hdfs://home/test/abc.txt      ----> s3://bucket/test/abc.txt

 

Highlighted
Posts: 790
Kudos: 23
Solutions: 10
Registered: ‎05-27-2014

Re: Unable to copy data from HDFS to S3

Hi @VenkateshB,

 

You may want to take a look at the doc here where provides some distcp examples:

https://www.cloudera.com/documentation/enterprise/6/latest/topics/cdh_admin_distcp_data_cluster_migr...

 

Thanks and hope it helps,

Li

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Announcements