Support Questions
Find answers, ask questions, and share your expertise

How to use S3a with Hive in HDP 2.5?

I am trying to create a table in S3 using HiveQL. I have added the following access key configs to HDFS core-site.xml and hive-site.xml:

fs.s3.awsAccessKeyId, fs.s3n.awsAccessKeyId, fs.s3a.awsAccessKeyId

fs.s3.awsSecretAccessKey, fs.s3n.awsSecretAccessKey, fs.s3a.awsSecretAccessKey

I have added what I believe are the relevant AWS jars to Hive's CP:

hive> add jar /usr/hdp/current/hadoop-client/hadoop-aws.jar;

hive> add jar /usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar;

hive> add jar /usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar;

Unfortunately, creating a table stored in s3a fails:

hive> create table test_backup_a stored as orc location 's3a://hwx-randy/test_backup_a' as select * from test;

...

Moving data to directory s3a://hwx-randy/test_backup_a

Failed with exception Unable to move source hdfs://dn0.dev:8020/apps/hive/warehouse/.hive-staging_hive_2016-10-07_15-14-34_668_7427003017223534386-1/-ext-10001 to destination s3a://hwx-randy/test_backup_a

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

3 REPLIES 3

Re: How to use S3a with Hive in HDP 2.5?

Worth noting: the above works for 's3n://' urls, but not 's3a://' or plain 's3://'

Re: How to use S3a with Hive in HDP 2.5?

New Contributor

Did you manage to get this working? btw, I think you need slightly different access and secret key i.e. fs.s3a.access.key fs.s3a.secret.key

But I'm not seeing HIVE even try to contact the S3A endpoint so wondering if you had success.

Re: How to use S3a with Hive in HDP 2.5?

Contributor

You need to specify "fs.s3a.access.key" and "fs.s3a.secret.key" . Also "fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" if it is not already present.

If it still throws exception in MoveTask, please share the /tmp/<user>/hive.log exception.