I am trying to create a table in S3 using HiveQL. I have added the following access key configs to HDFS core-site.xml and hive-site.xml:
fs.s3.awsAccessKeyId, fs.s3n.awsAccessKeyId, fs.s3a.awsAccessKeyId
fs.s3.awsSecretAccessKey, fs.s3n.awsSecretAccessKey, fs.s3a.awsSecretAccessKey
I have added what I believe are the relevant AWS jars to Hive's CP:
hive> add jar /usr/hdp/current/hadoop-client/hadoop-aws.jar;
hive> add jar /usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar;
hive> add jar /usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar;
Unfortunately, creating a table stored in s3a fails:
hive> create table test_backup_a stored as orc location 's3a://hwx-randy/test_backup_a' as select * from test;
Moving data to directory s3a://hwx-randy/test_backup_a
Failed with exception Unable to move source hdfs://dn0.dev:8020/apps/hive/warehouse/.hive-staging_hive_2016-10-07_15-14-34_668_7427003017223534386-1/-ext-10001 to destination s3a://hwx-randy/test_backup_a
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
Did you manage to get this working? btw, I think you need slightly different access and secret key i.e. fs.s3a.access.key fs.s3a.secret.key
But I'm not seeing HIVE even try to contact the S3A endpoint so wondering if you had success.
You need to specify "fs.s3a.access.key" and "fs.s3a.secret.key" . Also "fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" if it is not already present.
If it still throws exception in MoveTask, please share the /tmp/<user>/hive.log exception.