Python lib permissions cause Hue firstrun start failure on CDH6 deployments

dturner — Fri, 16 Sep 2022 14:32:36 GMT

Using Cloudera Altus Director 6.3 to deploy CDH 6.3 to AWS.

I submitted a similar post back in April, thinking that it had since been resolved - but apparently not. We are ramping up CDH6 deployments again since our initial testing in April and again are seeing Hue fail to start during first run of CDH6 deployments due to Python lib folders being too restrictive. Interestingly we do not see this issue when using the same CentOS 7.6 based AMI to deploy CDH 5.16.2 clusters. The Python lib permissions differ on between the CDH5 and CDH6 deployments:

CDH5:

ls -l /usr/lib/python2.7/site-packages/six*
-rw-r--r--. 1 root root 29664 Jan 2 2015 /usr/lib/python2.7/site-packages/six.py
-rw-r--r--. 1 root root 29708 Nov 20 2015 /usr/lib/python2.7/site-packages/six.pyc
-rw-r--r--. 1 root root 29708 Nov 20 2015 /usr/lib/python2.7/site-packages/six.pyo

/usr/lib/python2.7/site-packages/six-1.9.0-py2.7.egg-info:
total 16
-rw-r--r--. 1 root root 1 Nov 20 2015 dependency_links.txt
-rw-r--r--. 1 root root 1419 Nov 20 2015 PKG-INFO
-rw-r--r--. 1 root root 249 Nov 20 2015 SOURCES.txt
-rw-r--r--. 1 root root 4 Nov 20 2015 top_level.txt

CDH6:

ls -l /usr/lib/python2.7/site-packages/six*
-rw-r-----. 1 root root 32452 Aug 15 18:09 /usr/lib/python2.7/site-packages/six.py
-rw-r-----. 1 root root 31828 Aug 15 18:09 /usr/lib/python2.7/site-packages/six.pyc

/usr/lib/python2.7/site-packages/six-1.12.0.dist-info:
total 24
-rw-r-----. 1 root root 4 Aug 15 18:09 INSTALLER
-rw-r-----. 1 root root 1066 Aug 15 18:09 LICENSE
-rw-r-----. 1 root root 1940 Aug 15 18:09 METADATA
-rw-r-----. 1 root root 537 Aug 15 18:09 RECORD
-rw-r-----. 1 root root 4 Aug 15 18:09 top_level.txt
-rw-r-----. 1 root root 110 Aug 15 18:09 WHEEL

Again, we use the same AWS AMI for the CDH5 and 6 deployments. For CDH6 I have to manually fix the permissions on all the nodes prior to first run in order to avoid director declaring the deployment failed. This is really hampering our CDH6 rollout.

+ run_syncdb_and_migrate_subcommands

+ '[' 6 -ge 6 ']'

+ /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/bin/hue makemigrations --noinput

Traceback (most recent call last):

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/bin/hue", line 9, in <module>

    from pkg_resources import load_entry_point

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3241, in <module>

    @_call_aside

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside

    f(*args, **kwargs)

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set

    working_set = WorkingSet._build_master()

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 574, in _build_master

    ws = cls()

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in __init__

    self.add_entry(entry)

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 623, in add_entry

    for dist in find_distributions(entry, True):

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2056, in find_on_path

    for dist in factory(fullpath):

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2118, in distributions_from_metadata

    if len(os.listdir(path)) == 0:

OSError: [Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/six-1.12.0.dist-info'

+ /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/bin/hue migrate --fake-initial

Traceback (most recent call last):

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/bin/hue", line 9, in <module>

    from pkg_resources import load_entry_point

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3241, in <module>

    @_call_aside

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside

    f(*args, **kwargs)

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set

    working_set = WorkingSet._build_master()

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 574, in _build_master

    ws = cls()

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in __init__

    self.add_entry(entry)

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 623, in add_entry

    for dist in find_distributions(entry, True):

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2056, in find_on_path

    for dist in factory(fullpath):

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2118, in distributions_from_metadata

    if len(os.listdir(path)) == 0:

OSError: [Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/six-1.12.0.dist-info'

+ '[' dumpdata = runcpserver ']'

+ '[' syncdb = runcpserver ']'

+ '[' ldaptest = runcpserver ']'

+ exec /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/bin/hue runcpserver

Traceback (most recent call last):

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/bin/hue", line 9, in <module>

    from pkg_resources import load_entry_point

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3241, in <module>

    @_call_aside

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside

    f(*args, **kwargs)

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set

    working_set = WorkingSet._build_master()

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 574, in _build_master

    ws = cls()

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in __init__

    self.add_entry(entry)

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 623, in add_entry

    for dist in find_distributions(entry, True):

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2056, in find_on_path

    for dist in factory(fullpath):

  File "/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2118, in distributions_from_metadata

    if len(os.listdir(path)) == 0:

OSError: [Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/six-1.12.0.dist-info’

Re: Python lib permissions cause Hue firstrun start failure on CDH6 deployments

dturner — Fri, 16 Aug 2019 16:42:37 GMT

Some additional information...

We use Packer to build our images, and apply RedHat's CIS security policy for compliance reasons which sets a more restrictive umask in /etc/bashrc

if [ $UID -gt 199 ] && [ “/usr/bin/id -gn” = “/usr/bin/id -un” ]; then umask 027 else umask 022 fi

I'm thinking this doesn't effect our CDH5 deployments because those images already have the correct package version installed, however during CDH6 bootstrap, updated version are required and are installed using 027 umask resulting in no permissions for non-root user. Does the cluster bootstrap process assume umask 022 for non-root users?

Re: Python lib permissions cause Hue firstrun start failure on CDH6 deployments

dturner — Wed, 11 Sep 2019 20:15:42 GMT

We updated our Packer image build to skip the CIS rule for more restrictive umask (referenced above), after which Hue successfully started during cluster firstrun.

question Re: Python lib permissions cause Hue firstrun start failure on CDH6 deployments in Support Questions

Python lib permissions cause Hue firstrun start failure on CDH6 deployments

Re: Python lib permissions cause Hue firstrun start failure on CDH6 deployments

Re: Python lib permissions cause Hue firstrun start failure on CDH6 deployments