Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Configure HUE to access S3

Configure HUE to access S3

Master Collaborator

Hi all,

  I am fighting a little bit with hue safety valve ini properties, I cant make Hue and S3 work - i.e. regardless what I put into the hue_safety_valve.ini the HUE does not display the S3 browsing icon and I am not able to access the S3 file system. 

The cluster (cdh5.15) is kerberized, using Sentry. The node has a EC2 permission to access S3, so I assume this configuration should work:

 

[aws]
[[aws_accounts]]
[[[default]]]
allow_environment_credentials=true
region=eu-west-1

I am also able to browse S3 via HDFS (hdfs dfs -ls s3a://bucket/path) - and I dont have anything special set on HDFS. The user accessing the hue is a superuser and the "filebrowser" permission is granted to the user's group.

hue_s3_browsing.PNG

 

Any ideas what I missed in the configuration? I am having the same cluster running in older version (CDH5.13) and there this setting works.

Thanks

 

 

5 REPLIES 5

Re: Configure HUE to access S3

Guru
Have you tried to enable S3 Connector in CM, which will then enable for Hue automatically? Please refer to:

https://www.cloudera.com/documentation/enterprise/5-13-x/topics/admin_s3_config.html

Re: Configure HUE to access S3

Master Collaborator

No I havent tried, but on a different cluster I dont have S3 Connector service, and the connect to S3 works (i.e. there is an icon in Hue)

 

Btw all the nodes have IAM role granting access to s3 so I am not sure if I really need S3 connector:

 

 

I configured a new external account (IAM role based), set a name (any name) for the account and this triggered the process of adding S3 connector to the cluster. Then restarted the HUE service, and tried to log in, but the icon for the S3 browsing is still missing.

 

I am removing the S3 connector service from the cluster, as it did not helped. 

Re: Configure HUE to access S3

Master Collaborator

I checked the source code (running on CDH5.15) 

https://github.com/cloudera/hue/blob/cdh5-3.9.0_5.15.0/desktop/libs/aws/src/aws/conf.py

And it seems to display the S3 icon if the AWS key is present in the configuration or has_iam_metadata() returns True.

 

When I have added a access key and secret key properties (with empty values) then the S3 icon was displayed (but the browsing was not working, because I need IAM based access) 

 

I checked the IAM role and it seems to return the correct value:

 

 

>>> from boto import utils
>>> boto.utils.get_instance_metadata(timeout=1, num_retries=1)['iam']
{'info': {u'InstanceProfileArn': u'arn:aws:iam::8....... 

So 'iam' is in the metadata:

 

 

def has_iam_metadata():
  metadata = boto.utils.get_instance_metadata(timeout=1, num_retries=1)
  return 'iam' in metadata

I dont understand why it is not showing up the S3 browsing..

Re: Configure HUE to access S3

Master Collaborator
@EricLdo you have an idea or a hint what to change? Thanks
Highlighted

Re: Configure HUE to access S3

Guru
The switch is here:

https://github.com/cloudera/hue/blob/cdh5-3.9.0_5.15.0/desktop/core/src/desktop/templates/common_hea...

and is_s3_enabled is defined here:

https://github.com/cloudera/hue/blob/cdh5-3.9.0_5.15.0/desktop/core/src/desktop/settings.py#L36

However, I am not sure where aws.conf is from, might need to wait for other contributors' input here. I can't find the source of it to understand how the function works.