Permission errors loading Python modules: CDH6.2, CentOS7.6 FirstRun

[#1]

I've read the known issues for Hue in the Altus Director 6.2 docs, however according to the information in the docs, this shouldn't be applicable to CM6.2 and CentOS 7.6.  I'm seeing the following error when Hue attempts to start during the cluster's firstrun phase:

 /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue makemigrations --noinput
Traceback (most recent call last):
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue", line 9, in <module>
    from pkg_resources import load_entry_point
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 574, in _build_master
    ws = cls()
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in __init__
    self.add_entry(entry)
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 623, in add_entry
    for dist in find_distributions(entry, True):
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2033, in find_on_path
    for dist in factory(fullpath):
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2095, in distributions_from_metadata
    if len(os.listdir(path)) == 0:
OSError: [Errno 13] Permission denied: '/usr/lib64/python2.7/site-packages/wrapt-1.11.1.dist-info'
+ /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue migrate --fake-initial
Traceback (most recent call last):
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue", line 9, in <module>
    from pkg_resources import load_entry_point
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 574, in _build_master
    ws = cls()
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in __init__
    self.add_entry(entry)
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 623, in add_entry
    for dist in find_distributions(entry, True):
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2033, in find_on_path
    for dist in factory(fullpath):
  File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2095, in distributions_from_metadata
    if len(os.listdir(path)) == 0:
OSError: [Errno 13] Permission denied: '/usr/lib64/python2.7/site-packages/wrapt-1.11.1.dist-info'

Update:

I "solved" this by updating the site-package permissions under /usr/lib /usr/lib64, as they were locked down to root.  If I were to guess, some modules were pip installed without the -u switch, locking down access to root only.  I assume this should be handled by the deployment scripts when preparing the cluster nodes.

[#2]

I tried another deployment and hit the same errors.  All other cluster services start fine during first run.  Hue continues to fail to start due to insufficient permissions to load Python modules.  Pretty sure I could fix this if someone pointed me to where you handle the install of these Python modules.  Suprised no one else is reporting it.  

[#3]

Hi dturner,

I just attempted this and wasn't able to reproduce the issue. My cluster creation which included Hue succeeded using CM+CDH 6.2 with CentOS7.6.

In fact I couldn't even find the offending package (wrapt) anywhere on the Hue instance, so there's something different in our environments/installation.

Maybe more information about the enviornment/configuration would help narrow down the problem (whether your using preloaded/prextracted images, AWS or Azure, any special bootstrap scripts, etc)

[#4]

Thank you for your response.

We base our image on the stock CentOS AMI provided by AWS, with some security policies and tuning applied. Interestingly enough, we've deployed several 5.16.1 clusters using the same AMI.  This appears to be specific to CDH6.2.   I encounter this issue whether or not I try to include Hue in the Altus cluster config, or add the service post deployment from Cloudera Manager.   Wrapt is just the first module it complains about.  There are several others.  I ended up simply changing directory / file permissions for all dist-info, afterwhich I could add Hue service. 

chmod 755 /usr/lib64/python2.7/site-packages/*.dist-info
chmod -R 644 /usr/lib64/python2.7/site-packages/*.dist-info/
chmod 755 /usr/lib/python2.7/site-packages/*.dist-info
chmod -R 644 /usr/lib/python2.7/site-packages/*.dist-info/

I am using the latest version of Altus Director (6.2).  

[#5]

Thanks dturner, I'll request to move this to the Cloudera Manager forums. First run, and the scripts associated with it are handled by Cloudera Manager, so they may have more context on whether such hacks are needed.

[#6 · Accepted solution]

Update:  I rebuilt our image and no longer see this issue.  I still don't know why we didn't see this issue with Altus 2.8 using the same image, but I'll mark as resolved since we no longer see it with the new image.