Reply
Master
Posts: 326
Registered: ‎07-01-2015

Configure HUE to access S3

Hi all,

  I am fighting a little bit with hue safety valve ini properties, I cant make Hue and S3 work - i.e. regardless what I put into the hue_safety_valve.ini the HUE does not display the S3 browsing icon and I am not able to access the S3 file system. 

The cluster (cdh5.15) is kerberized, using Sentry. The node has a EC2 permission to access S3, so I assume this configuration should work:

 

[aws]
[[aws_accounts]]
[[[default]]]
allow_environment_credentials=true
region=eu-west-1

I am also able to browse S3 via HDFS (hdfs dfs -ls s3a://bucket/path) - and I dont have anything special set on HDFS. The user accessing the hue is a superuser and the "filebrowser" permission is granted to the user's group.

hue_s3_browsing.PNG

 

Any ideas what I missed in the configuration? I am having the same cluster running in older version (CDH5.13) and there this setting works.

Thanks

 

 

Cloudera Employee
Posts: 420
Registered: ‎03-23-2015

Re: Configure HUE to access S3

Have you tried to enable S3 Connector in CM, which will then enable for Hue automatically? Please refer to:

https://www.cloudera.com/documentation/enterprise/5-13-x/topics/admin_s3_config.html
Master
Posts: 326
Registered: ‎07-01-2015

Re: Configure HUE to access S3

No I havent tried, but on a different cluster I dont have S3 Connector service, and the connect to S3 works (i.e. there is an icon in Hue)

 

Btw all the nodes have IAM role granting access to s3 so I am not sure if I really need S3 connector:

 

 

I configured a new external account (IAM role based), set a name (any name) for the account and this triggered the process of adding S3 connector to the cluster. Then restarted the HUE service, and tried to log in, but the icon for the S3 browsing is still missing.

 

I am removing the S3 connector service from the cluster, as it did not helped. 

Master
Posts: 326
Registered: ‎07-01-2015

Re: Configure HUE to access S3

I checked the source code (running on CDH5.15) 

https://github.com/cloudera/hue/blob/cdh5-3.9.0_5.15.0/desktop/libs/aws/src/aws/conf.py

And it seems to display the S3 icon if the AWS key is present in the configuration or has_iam_metadata() returns True.

 

When I have added a access key and secret key properties (with empty values) then the S3 icon was displayed (but the browsing was not working, because I need IAM based access) 

 

I checked the IAM role and it seems to return the correct value:

 

 

>>> from boto import utils
>>> boto.utils.get_instance_metadata(timeout=1, num_retries=1)['iam']
{'info': {u'InstanceProfileArn': u'arn:aws:iam::8....... 

So 'iam' is in the metadata:

 

 

def has_iam_metadata():
  metadata = boto.utils.get_instance_metadata(timeout=1, num_retries=1)
  return 'iam' in metadata

I dont understand why it is not showing up the S3 browsing..

Highlighted
Master
Posts: 326
Registered: ‎07-01-2015

Re: Configure HUE to access S3

@EricLdo you have an idea or a hint what to change? Thanks
Cloudera Employee
Posts: 420
Registered: ‎03-23-2015

Re: Configure HUE to access S3

The switch is here:

https://github.com/cloudera/hue/blob/cdh5-3.9.0_5.15.0/desktop/core/src/desktop/templates/common_hea...

and is_s3_enabled is defined here:

https://github.com/cloudera/hue/blob/cdh5-3.9.0_5.15.0/desktop/core/src/desktop/settings.py#L36

However, I am not sure where aws.conf is from, might need to wait for other contributors' input here. I can't find the source of it to understand how the function works.
Announcements