Support Questions
Find answers, ask questions, and share your expertise

How to use S3a with Hive in HDP 2.5?

I am trying to create a table in S3 using HiveQL. I have added the following access key configs to HDFS core-site.xml and hive-site.xml:

fs.s3.awsAccessKeyId, fs.s3n.awsAccessKeyId, fs.s3a.awsAccessKeyId

fs.s3.awsSecretAccessKey, fs.s3n.awsSecretAccessKey, fs.s3a.awsSecretAccessKey

I have added what I believe are the relevant AWS jars to Hive's CP:

hive> add jar /usr/hdp/current/hadoop-client/hadoop-aws.jar;

hive> add jar /usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar;

hive> add jar /usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar;

Unfortunately, creating a table stored in s3a fails:

hive> create table test_backup_a stored as orc location 's3a://hwx-randy/test_backup_a' as select * from test;


Moving data to directory s3a://hwx-randy/test_backup_a

Failed with exception Unable to move source hdfs:// to destination s3a://hwx-randy/test_backup_a

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask


Worth noting: the above works for 's3n://' urls, but not 's3a://' or plain 's3://'

New Contributor

Did you manage to get this working? btw, I think you need slightly different access and secret key i.e. fs.s3a.access.key fs.s3a.secret.key

But I'm not seeing HIVE even try to contact the S3A endpoint so wondering if you had success.


You need to specify "fs.s3a.access.key" and "fs.s3a.secret.key" . Also "fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" if it is not already present.

If it still throws exception in MoveTask, please share the /tmp/<user>/hive.log exception.