Hive jobs are not running on s3n/impala failed to open hdfs files s3n://my_bucket

[#1]

Hi All,

I added  fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey properties in core,hdfs,hive-site.xml. I am able to run select * from tbl which is on s3 and got the resullt with beeline.But, when i run select count(*) from tbl, it's getting failed with following errors

Error: java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
       ........
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:251)
        ... 11 more
Caused by: java.io.IOException: s3n://rakeshs3 : 400 : Bad Request
        at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:453)
        at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:427)
                 ... 16 more
Caused by: org.jets3t.service.impl.rest.HttpException: 400 Bad Request
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:425)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279)
              ... 29 more

Impala Errors:

Failed to open HDFS file s3n://rakeshs3/tel.txt    Error(255): Unknown error 255

Thanks

[#2]

Hello Rakesh,

Unfortunately s3n filesystem scheme is not supported by Impala: http://www.cloudera.com/documentation/enterprise/latest/topics/impala_s3.html#s3_restrictions

If you set up s3a, please look at the recommended settings at "Best Practices for Using Impala with S3" section on the same page.

Regards

Miklos Szurap

Customer Operations Engineer

[#3]

Thanks Miklos..

What about hive MR jobs ?

[#4]

Hi,

If you see the same for Hive/MapReduce jobs, then please can you confirm that the table in question has many partitions? We have seen similar problems for tables where it contained over thousand or ten thousands of partitions.

[#5 · Accepted solution]

MR/Hive Jobs Issue resolved by replacing old jars with new version and switched to s3a. Below are the jars that are replaced.

jets3t jar

aws-java-sdk jars

jackson jars