question Re: Hive jobs are not running on s3n/impala failed to open hdfs files s3n://my_bucket in Archives of Support Questions (Read Only)

Hive jobs are not running on s3n/impala failed to open hdfs files s3n://my_bucket

RakeshE — Fri, 16 Sep 2022 10:55:56 GMT

Hi All,

I added fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey properties in core,hdfs,hive-site.xml. I am able to run select * from tbl which is on s3 and got the resullt with beeline.But, when i run select count(*) from tbl, it's getting failed with following errors

Error: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
........
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:251)
... 11 more
Caused by: java.io.IOException: s3n://rakeshs3 : 400 : Bad Request
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:453)
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:427)
... 16 more
Caused by: org.jets3t.service.impl.rest.HttpException: 400 Bad Request
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:425)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279)
... 29 more

Impala Errors:

Failed to open HDFS file s3n://rakeshs3/tel.txt Error(255): Unknown error 255

Thanks

Re: Hive jobs are not running on s3n/impala failed to open hdfs files s3n://my_bucket

mszurap — Mon, 23 Jan 2017 12:35:42 GMT

Hello Rakesh,

Unfortunately s3n filesystem scheme is not supported by Impala: http://www.cloudera.com/documentation/enterprise/latest/topics/impala_s3.html#s3_restrictions

If you set up s3a, please look at the recommended settings at "Best Practices for Using Impala with S3" section on the same page.

Regards

Miklos Szurap

Customer Operations Engineer

Re: Hive jobs are not running on s3n/impala failed to open hdfs files s3n://my_bucket

RakeshE — Mon, 23 Jan 2017 12:46:31 GMT

Thanks Miklos..

What about hive MR jobs ?

Re: Hive jobs are not running on s3n/impala failed to open hdfs files s3n://my_bucket

mszurap — Mon, 23 Jan 2017 12:51:16 GMT

Hi,

If you see the same for Hive/MapReduce jobs, then please can you confirm that the table in question has many partitions? We have seen similar problems for tables where it contained over thousand or ten thousands of partitions.

Re: Hive jobs are not running on s3n/impala failed to open hdfs files s3n://my_bucket

RakeshE — Wed, 01 Feb 2017 06:17:00 GMT

MR/Hive Jobs Issue resolved by replacing old jars with new version and switched to s3a. Below are the jars that are replaced.

jets3t jar

aws-java-sdk jars

jackson jars