Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

YARN Failed to start in CDH 5.7.1 (Cloudera Manager), Permission Denied to Supervisor.conf

YARN Failed to start in CDH 5.7.1 (Cloudera Manager), Permission Denied to Supervisor.conf

New Contributor

Hi,

 

I have installed Cloudera 5.7.1 using CentOS 7. Issue is that resource manager is not starting in YARN MR2.

 

Error is permission denied, below is the snippet from stderr.log file.

At present supervisor.conf file is being created as root user with read permission only to root, that might be the reason, however, i am not sure why would that be the case.

 

Please help.

 

+ ADD_TO_CP=/usr/share/cmf/lib/plugins/event-publish-5.7.1-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.7.1.jar:
+ [[ -n '' ]]
+ eval 'OLD_VALUE=$HADOOP_CLASSPATH'
++ OLD_VALUE=
+ NEW_VALUE=/usr/share/cmf/lib/plugins/event-publish-5.7.1-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.7.1.jar:
+ export HADOOP_CLASSPATH=/usr/share/cmf/lib/plugins/event-publish-5.7.1-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.7.1.jar
+ HADOOP_CLASSPATH=/usr/share/cmf/lib/plugins/event-publish-5.7.1-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.7.1.jar
+ set -x
+ replace_conf_dir
+ find /run/cloudera-scm-agent/process/207-yarn-RESOURCEMANAGER -type f '!' -path '/run/cloudera-scm-agent/process/207-yarn-RESOURCEMANAGER/logs/*' '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/207-yarn-RESOURCEMANAGER#g' '{}' ';'
Can't open /run/cloudera-scm-agent/process/207-yarn-RESOURCEMANAGER/supervisor.conf: Permission denied.
+ perl -pi -e 's#{{CGROUP_GROUP_CPU}}##g' /run/cloudera-scm-agent/process/207-yarn-RESOURCEMANAGER/yarn-site.xml

 

Thanks.

 

6 REPLIES 6

Re: YARN Failed to start in CDH 5.7.1 (Cloudera Manager), Permission Denied to Supervisor.conf

New Contributor

Hi,

 

I have the same problem on RHEL 7.2 new install. Let's see what Cloduera support has to say.

 

Steven

Re: YARN Failed to start in CDH 5.7.1 (Cloudera Manager), Permission Denied to Supervisor.conf

New Contributor

same issue with Impala 

cloudera 5.7.2 using rhel 7.1

 

 

+ find /run/cloudera-scm-agent/process/48-impala-CATALOGSERVER -type f '!' -path '/run/cloudera-scm-agent/process/48-impala-CATALOGSERVER/logs/*' '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/48-impala-CATALOGSERVER#g' '{}' ';'
Can't open /run/cloudera-scm-agent/process/48-impala-CATALOGSERVER/supervisor.conf: Permission denied.
+ make_scripts_executable
+ find /run/cloudera-scm-agent/process/48-impala-CATALOGSERVER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/48-impala-CATALOGSERVER#g' /run/cloudera-scm-agent/process/48-impala-CATALOGSERVER/impala-conf/catalogserver_flags
+ perl -pi -e 's#{{CGROUP_ROOT_CPU}}#/sys/fs/cgroup/cpu,cpuacct#g' /run/cloudera-scm-agent/process/48-impala-CATALOGSERVER/impala-conf/catalogserver_flags
+ '[' -f /run/cloudera-scm-agent/process/48-impala-CATALOGSERVER/impala-conf/.htpasswd ']'
+ chmod 600 /run/cloudera-scm-agent/process/48-impala-CATALOGSERVER/impala-conf/.htpasswd
+ false
+ export IMPALA_BIN=/usr/lib/impala/sbin-retail
+ IMPALA_BIN=/usr/lib/impala/sbin-retail
+ [[ true = '' ]]
+ '[' impalad = catalogd ']'
+ '[' statestore = catalogd ']'
+ '[' catalogd = catalogd ']'
+ exec /usr/lib/impala/../../bin/catalogd --flagfile=/run/cloudera-scm-agent/process/48-impala-CATALOGSERVER/impala-conf/catalogserver_flags
Redirecting stderr to /var/log/catalogd/catalogd.ERROR

 

Re: YARN Failed to start in CDH 5.7.1 (Cloudera Manager), Permission Denied to Supervisor.conf

New Contributor

I'm having the same problem, cloudera manager is creating the supervisor.conf files with ownership root:root.

 

Using Cloudera 5.9 and CentOS 7.2.1511

Re: YARN Failed to start in CDH 5.7.1 (Cloudera Manager), Permission Denied to Supervisor.conf

New Contributor
Actually ownership root:root of that file is normal... but why is it failing when starting services?

Re: YARN Failed to start in CDH 5.7.1 (Cloudera Manager), Permission Denied to Supervisor.conf

New Contributor

Not sure of the reason, everytime supervisor.conf is created it should be created with read permission to all the users, however, it is currently being created with read permission to only root, that might be the reason. 

Since, cloudera manager services can run with any user with sudo root privilages, it is still failing and that was the reason for this post seeking clarification on the same.

Re: YARN Failed to start in CDH 5.7.1 (Cloudera Manager), Permission Denied to Supervisor.conf

New Contributor

I have had this problem on multiple clusters. Sometimes, the error in the actual log file in /var/log/hadoop-yarn/ shows an error relating to folder permissions which I solved by setting the perm on that folder with chmod -R a+rwx /folderURI/ . The ownership by root of the files in question is a problem but not the root cause. I traced the actual root cause by observing the role log for YARN and seeing that each time it started and crashed it was trying to run an old job. This was due to child folders in /var/lib/hadoop-yarn/yarn-nm-recovery/   :   nm-aux-services, and yarn-nm-state. I moved these to /tmp/ and was able to start the NodeManager. 

 

Next steps:

1) reognize the permission denied is a real error but not the root cause

2) possibly overcome problem by setting global perms on parent directory

3) read yarn log and find errors or evidence of repeated attempts to recover from a bad state

4) Cloudera/Apache open community should mark this as multiple bugs: a) poor error message misleading to actual root cause, b) setting file owner as root which should not occur in anycase for security. It should be owned by yarn.