Reply
New Contributor
Posts: 5
Registered: ‎01-27-2019

CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied

[ Edited ]

Hey All,

 

Running into:

 

Can't open /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied.

 on a vanilla 6.2 installation.

 

namei output (from a non-namenode server):

 

[root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# namei -l /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status
f: /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status
dr-xr-xr-x root root /
drwxr-xr-x root root var
lrwxrwxrwx root root run -> ../run
dr-xr-xr-x root root ..
drwxr-xr-x root root run
drwxr-x--x root root cloudera-scm-agent
drwxr-x--x root root process
drwxr-x--x cloudera-scm cloudera-scm 114-cloudera-mgmt-EVENTSERVER
-rw------- root root supervisor_status
[root@cm-se02 114-cloudera-mgmt-EVENTSERVER]#

When?

 

During installation of:

 

Add Cloudera Management Service Service

 

LOGS:

 

+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_181-cloudera ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
Can't open /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied.
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_181-cloudera '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_181-cloudera/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x

 

Already added the hosts earlier via this CM 6.2 to a vanilla cluster.   Anyway to get around the above error?

 

Edit:

Seems like supervisord creates the files as root but cloudera-scm-agent tries to read them as cloudera-scm user.  That won't work.  

 

[root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# find /var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR -type f '!' -path '/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name cmon.keytab '!' -name '*jceks' -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR#g' '{}' ';'
[root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# echo $?
0
[root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# sudo su - cloudera-scm
Last login: Sun Mar 31 00:42:49 EDT 2019 on pts/0
-bash-4.2$
-bash-4.2$
-bash-4.2$
-bash-4.2$ find /var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR -type f '!' -path '/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name cmon.keytab '!' -name '*jceks' -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR#g' '{}' ';'
Can't open /var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR/supervisor_status: Permission denied.
-bash-4.2$ who am i
root     pts/0        2019-04-14 11:06 (192.168.0.142)
-bash-4.2$ whoami
cloudera-scm
-bash-4.2$

 

Additional analysis:

 

[root@cm-se02 cloudera-scm-agent]# sudo su - cloudera-scm
Last login: Sun Apr 14 11:41:29 EDT 2019 on pts/0
-bash-4.2$ cd /var/run/cloudera-scm-agent/
-bash-4.2$
-bash-4.2$ ls -altri
ls: cannot open directory .: Permission denied
-bash-4.2$ cd process
-bash-4.2$ pwd
/var/run/cloudera-scm-agent/process
-bash-4.2$ cd 114-cloudera-mgmt-EVENTSERVER
-bash-4.2$ ls -altri supervisor
supervisor.conf    supervisor_status
-bash-4.2$ ls -altri supervisor_status
714067 -rw-------. 1 root root 656 Apr 14 11:01 supervisor_status
-bash-4.2$ cat supervisor_status
cat: supervisor_status: Permission denied
-bash-4.2$ pwd
/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER
-bash-4.2$

 

Thx,

TK

Posts: 1,045
Topics: 1
Kudos: 263
Solutions: 131
Registered: ‎04-22-2014

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

Hi @TCloud ,

 

Thank you for all the information.

It seems that we may have missed excluding the supservisor_status file from the exclusion patterns listed in the replacement function in cloudera-config.sh.

 

This error shouldn't be fatal, though, so if you are seeing a problem in your servers, it is likely something else causing the issue.

 

I'll file a Jira with the Cloudera Manager team to see about excluding it.  The processes or the shell scripts that start the processes do not need to read the supervisor_status file; only the CM agent reads/writes and that the agent runs as root.

 

Regards,

 

Ben

New Contributor
Posts: 5
Registered: ‎01-27-2019

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

Hey bgooley,

 

The fatal issue turned out to be the SSL Certs issue.  I've created the certs and distributed them throughout the cluster. 

 

However found it odd that one service creates supervisor_status but another cloudera service can't read it and so posted about it.  Glad  you found it helpful. 

 

Cheers,
TK

Posts: 1,045
Topics: 1
Kudos: 263
Solutions: 131
Registered: ‎04-22-2014

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

@TCloudIt is great to hear that you isolated and resolved the problem.  The supervisor_status file shouldn't need the config replacement at all so I opened Cloudera internal Jira OPSAPS-50270 to request we add it to the list of patterns that are excluded from replacing the placeholder in config files with the path to the process directory.  That should address the issue so the Permission Denied message doesn't occur any more.

 

Thanks for bringing it up so we can fix it!

 

Ben

Announcements