Support Questions
Find answers, ask questions, and share your expertise

NoAuthWithAWSException while running create database in hive and hdfs ls

Solved Go to solution
Highlighted

NoAuthWithAWSException while running create database in hive and hdfs ls

New Contributor

While trying to create a database through hive in Hue or using jdbc, I reveive the following error.

Error while compiling statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.nio.file.AccessDeniedException: [bucket]: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: Missing required credential provisioning for Cloud Access Broker. It is expected that keys and secrets be provisioned as aliases or that Cloud Access Broker be running on a node with an Instance Profile attached.

If I ssh into a node and run hdfs dfs -ls s3a://[bucket-name]/ I also see

20/10/29 19:42:47 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
Oct 29, 2020 7:42:47 PM org.apache.knox.gateway.shell.KnoxSession createClient
INFO: Using default JAAS configuration
20/10/29 19:42:48 INFO Configuration.deprecation: No unit for fs.s3a.connection.request.timeout(0) assuming SECONDS
20/10/29 19:42:48 ERROR common.DefaultRequestExecutor: Error executing request: org.apache.knox.gateway.shell.ErrorResponse: https://[broker-name]:8444/gateway/aws-cab/cab/api/v1/credentials: HTTP/1.1 500 Internal Server Error
20/10/29 19:42:48 ERROR idbroker.AbstractIDBClient: Cloud Access Broker response: {
  "error": "Missing required credential provisioning for Cloud Access Broker. It is expected that keys and secrets be provisioned as aliases or that Cloud Access Broker be running on a node with an Instance Profile attached."
}

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NoAuthWithAWSException while running create database in hive and hdfs ls

New Contributor

This was resolved by enabling IMDSv1 on the idbroker ec2 instance. 
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html 

aws ec2 modify-instance-metadata-options \
    --instance-id [idbroker instance id] \
    --http-tokens optional \
    --http-endpoint enabled

View solution in original post

1 REPLY 1
Highlighted

Re: NoAuthWithAWSException while running create database in hive and hdfs ls

New Contributor

This was resolved by enabling IMDSv1 on the idbroker ec2 instance. 
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html 

aws ec2 modify-instance-metadata-options \
    --instance-id [idbroker instance id] \
    --http-tokens optional \
    --http-endpoint enabled

View solution in original post