cloudera does not support access to s3 within eu/frankfurt: Aws Signature v4 not supported?

[#1]

Hey guys.

We are installed a cluster on top of aws infrastructure. 

AWS introduced a new authentication methods (signature v4) in eu/frankfurt for accessing S3/s3n/s3a.

Region EU only allows signature v4 and not the old v2 as other regions (us).

http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#ConstructingTheAuthentication...

We are facing the problem by reading data from S3 into a pig-job and facing the following error-code:

"The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256"

The problem is the jets3t.jar package that comes with hadoop-common. Cloudera is delivered with version 0.9.0.  

Jets3t supports the new signature v4 in jets4t.jar v0.9.3.

http://www.jets3t.org/RELEASE_NOTES.html

How can we solve that problem without going away from cloudera?

1. Replace the jets3t.jar libs? okay how is that possbile?

2. Saying hadoop s3 requests to use different signatures (this is just a information in the REST header)

3. Compile hadoop with the latest jets3t.jar

4. Partials

Would be great if someone can help us out because that is a really big problem to us

Cheers

Kay

[#2]

Does using the (AMZN provided) s3a:// filesystem instead give the same
problem? Have you tried that already?

[#3]

Hi. Thanks for the fast reply!!!

okay. we tried the s3a but we get a bad-request response. We are contacting aws support why that happened.

Anyway, we still need the s3 protocol, due to the fact that we have really  a lot of data.

Do you have some solutions how we can solve that problem? Is it possible to replace the jar file jets3t-0.9.0.jar with the latest one?

Cheers

[#4]

Has anyone been able to fix this issue?

[#5]

unfortunately not.

You can sync the data to the us standard region bucket. (region sync s3) and then you are able to read it.

[#6]

Hi, 

signature V4 support using s3a in CDH still seems to be broken (up to CDH 5.5.1). To fix this we created a fork of hadoop-aws that

• sets the system property "com.amazonaws.services.s3.enableV4" to enable V4 support
• includes the requried joda time classes in the shaded hadoop-aws jar

With those modifications s3a with Signature V4 is enabled if the property "fs.s3a.endpoint" is set. For accessing buckets in Frankfurt use fs.s3a.endpoint=s3.eu-central-1.amazonaws.com.

Best,

Helmut

[#7]

I believe the fix was included in JIRA https://issues.apache.org/jira/browse/HADOOP-12269, which is part of CDH5.5.2 see: https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_552.html

[#8]

Hi

How/Where to update the value of "com.amazonaws.services.s3.enableV4" on cloudera  to resolve the s3 connectivity issue.

Please help.

[#9]

The switch is not necessary to manually enable, as it is enabled by default for required S3 operations based on region automatically.

What version of CDH are you using to connect to S3?
What connector of S3 are you using (we recommend using s3a:// over anything else)?
What is your command and what is your received error?
How are you providing your configuration?

[#10]

Hi Following are the answers of questions

What version of CDH are you using to connect to S3?
[5.7.0]

What connector of S3 are you using (we recommend using s3a:// over anything
else)?
[tried with both s3n:// and s3a://]
What is your command and what is your received error?

> I am using spark-shell

val hadoopConf = sc.hadoopConfiguration;
hadoopConf.set("fs.s3a.access.key", );
hadoopConf.set("fs.s3a.secret.key", );
hadoopConf.set("fs.s3.access.key", );
hadoopConf.set("fs.s3.secret.key", );
hadoopConf.set("fs.s3n.access.key", );
hadoopConf.set("fs.s3n.secret.key", );


var df = sqlContext.read.parquet("s3a:///)

df.show();

Exception :

16/08/25 13:46:24 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0
(TID 2, hadoop03.monetization.dev.ggn1.ops.snapdeal.io):
com.cloudera.com.amazonaws.AmazonClientException: Unable to load AWS
credentials from any provider in the chain
at
com.cloudera.com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
at
com.cloudera.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3779)
at
com.cloudera.com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1107)
at
com.cloudera.com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:322)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2696)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2733)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2715)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:385)
at
parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:162)
at
parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:145)
at
org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.(SqlNewHadoopRDD.scala:180)
at
org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:126)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)