Reply
New Contributor
Posts: 1
Registered: ‎09-25-2013

Problems with Hive and S3 .

I am having issues accessing S3 with hive.  I have my keys and ID under hive-site.xml and i am able to see them when I run

hive> set;  

 

I get a list of all the attributes for hive but when I try to access s3 to create a table i get the following error.

 

FAILED: Error in metadata: MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).)

 

I even set the fs.s3 and fs.s3n attributes manually using the set command under hive. 

 

Still no luck .  Any suggestions or ideas

 

Thanks so much

Posts: 1,567
Kudos: 289
Solutions: 240
Registered: ‎07-31-2013

Re: Problems with Hive and S3 .

Please try placing the properties in your Hadoop config's core-site.xml.

If you use Cloudera Manager, add the fs.s3.* and fs.s3n.* properties in XML form to CM -> HDFS -> Configuration -> "core-site.xml cluster wide safety valve" and save plus do a CM -> Actions -> Deploy Client Configuration.
Backline Customer Operations Engineer
New Contributor
Posts: 1
Registered: ‎09-12-2014

Re: Problems with Hive and S3 .

Do you copy the entire block of XML into the Value text box? Like:

 

<property>
<name>fs.s3.awsAccessKeyId</name>
<value>asdfasd</value>
</property>

<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>asdfasdf</value>
</property>

 

I did that, deployed the client configuration, and then went to /etc/hadoop/conf/core-site.xml and there were no changes. STill can't see S3 buckets.

New Contributor
Posts: 2
Registered: ‎07-26-2017

Re: Problems with Hive and S3 .

Hi,

Do you able to resolve this issue. I understand that its been a long time, since you have posted this. Now I am facing the same Issue, your suggession for resolving this would be helpful for me.

 

Thanks in advance.

New Contributor
Posts: 1
Registered: ‎10-05-2017

Re: Problems with Hive and S3 .

We are also facing the same issue , we have already added aprameters in  hdfs-site.xml and core-site.xml  but still getting the error while runing create external table statement.

message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL,

 

any help on this would be realy helpful

 

Thanks

Jayendran

Highlighted
New Contributor
Posts: 2
Registered: ‎07-26-2017

Re: Problems with Hive and S3 .

Hi Jayendran,
I did not find solution yet, however after speaking with few of my friends
found that the configuration to be made in EMR cluster only and not outside
Amazon. I speculate the same, but not verified at my end. In case if you
could resolve this issue after working out in EMR rather than cloudera,
then let me know.

Regards,
Leela Prasad
Announcements