We are installed a cluster on top of aws infrastructure.
AWS introduced a new authentication methods (signature v4) in eu/frankfurt for accessing S3/s3n/s3a.
Region EU only allows signature v4 and not the old v2 as other regions (us).
We are facing the problem by reading data from S3 into a pig-job and facing the following error-code:
"The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256"
The problem is the jets3t.jar package that comes with hadoop-common. Cloudera is delivered with version 0.9.0.
Jets3t supports the new signature v4 in jets4t.jar v0.9.3.
How can we solve that problem without going away from cloudera?
1. Replace the jets3t.jar libs? okay how is that possbile?
2. Saying hadoop s3 requests to use different signatures (this is just a information in the REST header)
3. Compile hadoop with the latest jets3t.jar
Would be great if someone can help us out because that is a really big problem to us
Hi. Thanks for the fast reply!!!
okay. we tried the s3a but we get a bad-request response. We are contacting aws support why that happened.
Anyway, we still need the s3 protocol, due to the fact that we have really a lot of data.
Do you have some solutions how we can solve that problem? Is it possible to replace the jar file jets3t-0.9.0.jar with the latest one?
You can sync the data to the us standard region bucket. (region sync s3) and then you are able to read it.
signature V4 support using s3a in CDH still seems to be broken (up to CDH 5.5.1). To fix this we created a fork of hadoop-aws that
With those modifications s3a with Signature V4 is enabled if the property "fs.s3a.endpoint" is set. For accessing buckets in Frankfurt use fs.s3a.endpoint=s3.eu-central-1.amazonaws.com.
I believe the fix was included in JIRA https://issues.apache.org/jira/browse/HADOOP-12269, which is part of CDH5.5.2 see: https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_552.html
How/Where to update the value of "com.amazonaws.services.s3.enableV4" on cloudera to resolve the s3 connectivity issue.