Permissions on /var/lib/<directory> causing services to fail.

[#1]

I am setting up a CDH (5.8) Cluster with Cloudera Manager and Parcels. Most of the services are installed and running fine with an exception of HDFS HTTPFS role and Oozie. I am using a custom parcel directory on the agents  and also I am running SELinux in Enforcing mode (Please dont ask me to put it to permissive 🙂 ).

Both has similar isssue where its trying to move some files (related to tomcat) from the parcel's directory to /var/lib/<directory>.

When I checked the permissions of the directories in /var/lib I could see that most of the directories are created without any permissions set on them.

$ ls -l /var/lib/ | grep ^d-
d---------.  2 root         root            6 Jan 13 00:28 flume-ng
d---------.  2 root         root            6 Jan 13 00:28 hadoop-hdfs
d---------.  2 root         root            6 Jan 13 00:28 hadoop-httpfs
d---------.  2 root         root            6 Jan 13 00:28 hadoop-kms
d---------.  2 root         root            6 Jan 13 00:28 hadoop-mapreduce
d---------.  2 root         root            6 Jan 13 00:28 hadoop-yarn
d---------.  2 root         root            6 Jan 13 00:28 hbase
d---------.  2 root         root            6 Jan 13 00:28 hive
d---------.  2 root         root            6 Jan 13 00:28 impala
d---------.  2 root         root            6 Jan 13 00:29 kafka
d---------.  2 root         root            6 Jan 13 00:28 llama
d---------.  2 root         root            6 Jan 13 00:28 oozie
d---------.  2 root         root            6 Jan 13 00:28 sentry
d---------.  2 root         root            6 Jan 13 00:28 solr
d---------.  2 root         root            6 Jan 13 00:28 spark
d---------.  2 root         root            6 Jan 13 00:28 sqoop
d---------.  2 root         root            6 Jan 13 00:28 sqoop2
d---------.  2 root         root            6 Jan 13 00:28 zookeeper

Is this expected?

For Oozie, I changed the permission of /var/lib/oozie to 700 and changed the ownership to oozie:oozie fixed the issue.

What would be the reason for this permission issue and how could be this fixed.

Thanks in advance.

[#2 · Accepted solution]

Hello,

This is not an expected condition, it would appear as though you are using a custom umask or SELinux is blocking certain operations performed by the agent. Please note that our documentation specifically states that you must disable SELinux before the deployment of CDH.

If you intend to use SELinux you will need to design your own SELinux profiles that secure the envirnoment and allow for the many different types of operations performed across all host in your environment.

http://www.cloudera.com/documentation/enterprise/latest/topics/install_cdh_dependencies.html

http://www.cloudera.com/documentation/enterprise/latest/topics/install_cdh_disable_selinux.html

Trying to use SELinux in the hadoop stack is tedious because there are hundreds of tools and widgets which require complex SELinux Policies. Policies which must be revisited frequently when new features or changes occur in the platform.

That means that you can make Hadoop work with SELinux, it's not impossible but it requires a high level of continuous effort from the organization because no single vendor has created nor provides standardized policies which work in all cases. In fact if the apache foundation made standardized policies they still would not be entirely valid for environments managed by Cloudera Manager depending on the specificity of the policies.

Unfortunately I do not have a complete list of directories and permissions handy but here is example output from a lab environment.

# namei -l /var/lib/* |egrep 'zookeeper|sqoop2|sqoop|spark|solr|sentry|oozie|hadoop'
f: /var/lib/hadoop-hdfs
drwxr-xr-x hdfs hdfs hadoop-hdfs
f: /var/lib/hadoop-httpfs
drwxr-xr-x httpfs httpfs hadoop-httpfs
f: /var/lib/hadoop-kms
drwxr-xr-x kms kms hadoop-kms
f: /var/lib/hadoop-mapreduce
drwxr-xr-x mapred mapred hadoop-mapreduce
f: /var/lib/hadoop-yarn
drwxr-xr-x yarn yarn hadoop-yarn
f: /var/lib/oozie
drwxr-xr-x oozie oozie oozie
f: /var/lib/sentry
drwxr-xr-x sentry sentry sentry
f: /var/lib/solr
drwxr-xr-x solr solr solr
f: /var/lib/spark
drwxr-xr-x spark spark spark
f: /var/lib/sqoop
drwxr-xr-x sqoop sqoop sqoop
f: /var/lib/sqoop2
drwxr-xr-x sqoop2 sqoop2 sqoop2
f: /var/lib/zookeeper
drwxr-xr-x zookeeper zookeeper zookeeper

[#3]

Thank you @lhebert. Looks like it was actually caused by SELinux, and we now have custom selinux policies to overcome this issue.