Reply
Contributor
Posts: 65
Registered: ‎05-19-2016

distcp from cluster to s3

I need to copy hdfs data from cluster to s3.

 

If I use s3a, I get:

 

 

 WARN s3a.S3AFileSystem: Client: Amazon S3 error 400: 400 Bad Request; Bad Request (retryable)

 

If I instead try with S3, it says:

The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

 

My bucket is in Asia/Mumbai region and I tried creating one in US east. Nothing changed.

 

Posts: 388
Topics: 11
Kudos: 60
Solutions: 34
Registered: ‎09-02-2016

Re: distcp from cluster to s3

@sim6

 

A similar issue fix by applying attached patch in this link. It 'may' help you

 

https://issues.apache.org/jira/browse/HADOOP-13324

 

 

Announcements
New solutions