Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied

CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied

Explorer

Hey All,

 

Running into:

 

Can't open /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied.

 on a vanilla 6.2 installation.

 

namei output (from a non-namenode server):

 

[root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# namei -l /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status
f: /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status
dr-xr-xr-x root root /
drwxr-xr-x root root var
lrwxrwxrwx root root run -> ../run
dr-xr-xr-x root root ..
drwxr-xr-x root root run
drwxr-x--x root root cloudera-scm-agent
drwxr-x--x root root process
drwxr-x--x cloudera-scm cloudera-scm 114-cloudera-mgmt-EVENTSERVER
-rw------- root root supervisor_status
[root@cm-se02 114-cloudera-mgmt-EVENTSERVER]#

When?

 

During installation of:

 

Add Cloudera Management Service Service

 

LOGS:

 

+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_181-cloudera ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
Can't open /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied.
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_181-cloudera '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_181-cloudera/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x

 

Already added the hosts earlier via this CM 6.2 to a vanilla cluster.   Anyway to get around the above error?

 

Edit:

Seems like supervisord creates the files as root but cloudera-scm-agent tries to read them as cloudera-scm user.  That won't work.  

 

[root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# find /var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR -type f '!' -path '/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name cmon.keytab '!' -name '*jceks' -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR#g' '{}' ';'
[root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# echo $?
0
[root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# sudo su - cloudera-scm
Last login: Sun Mar 31 00:42:49 EDT 2019 on pts/0
-bash-4.2$
-bash-4.2$
-bash-4.2$
-bash-4.2$ find /var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR -type f '!' -path '/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name cmon.keytab '!' -name '*jceks' -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR#g' '{}' ';'
Can't open /var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR/supervisor_status: Permission denied.
-bash-4.2$ who am i
root     pts/0        2019-04-14 11:06 (192.168.0.142)
-bash-4.2$ whoami
cloudera-scm
-bash-4.2$

 

Additional analysis:

 

[root@cm-se02 cloudera-scm-agent]# sudo su - cloudera-scm
Last login: Sun Apr 14 11:41:29 EDT 2019 on pts/0
-bash-4.2$ cd /var/run/cloudera-scm-agent/
-bash-4.2$
-bash-4.2$ ls -altri
ls: cannot open directory .: Permission denied
-bash-4.2$ cd process
-bash-4.2$ pwd
/var/run/cloudera-scm-agent/process
-bash-4.2$ cd 114-cloudera-mgmt-EVENTSERVER
-bash-4.2$ ls -altri supervisor
supervisor.conf    supervisor_status
-bash-4.2$ ls -altri supervisor_status
714067 -rw-------. 1 root root 656 Apr 14 11:01 supervisor_status
-bash-4.2$ cat supervisor_status
cat: supervisor_status: Permission denied
-bash-4.2$ pwd
/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER
-bash-4.2$

 

Thx,

TK

9 REPLIES 9

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

Super Guru

Hi @TCloud ,

 

Thank you for all the information.

It seems that we may have missed excluding the supservisor_status file from the exclusion patterns listed in the replacement function in cloudera-config.sh.

 

This error shouldn't be fatal, though, so if you are seeing a problem in your servers, it is likely something else causing the issue.

 

I'll file a Jira with the Cloudera Manager team to see about excluding it.  The processes or the shell scripts that start the processes do not need to read the supervisor_status file; only the CM agent reads/writes and that the agent runs as root.

 

Regards,

 

Ben

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

Explorer

Hey bgooley,

 

The fatal issue turned out to be the SSL Certs issue.  I've created the certs and distributed them throughout the cluster. 

 

However found it odd that one service creates supervisor_status but another cloudera service can't read it and so posted about it.  Glad  you found it helpful. 

 

Cheers,
TK

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

Super Guru

@TCloudIt is great to hear that you isolated and resolved the problem.  The supervisor_status file shouldn't need the config replacement at all so I opened Cloudera internal Jira OPSAPS-50270 to request we add it to the list of patterns that are excluded from replacing the placeholder in config files with the path to the process directory.  That should address the issue so the Permission Denied message doesn't occur any more.

 

Thanks for bringing it up so we can fix it!

 

Ben

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

New Contributor

Hi

 

I am getting the same permission denied error while installing the essential services on the cluster.

Could you let me know what I am doing wrong?

 

Can't open /var/run/cloudera-scm-agent/process/67-hdfs-NAMENODE-format/supervisor_status: Permission denied.
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/67-hdfs-NAMENODE-format -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ '[' DATANODE_MAX_LOCKED_MEMORY '!=' '' ']'
+ ulimit -l
+ export HADOOP_IDENT_STRING=hdfs
+ HADOOP_IDENT_STRING=hdfs
+ '[' -n '' ']'
+ '[' mkdir '!=' format-namenode ']'
+ acquire_kerberos_tgt hdfs.keytab
+ '[' -z hdfs.keytab ']'
+ KERBEROS_PRINCIPAL=
+ '[' '!' -z '' ']'
+ '[' -n '' ']'
+ '[' validate-writable-empty-dirs = format-namenode ']'
+ '[' file-operation = format-namenode ']'
+ '[' bootstrap = format-namenode ']'
+ '[' failover = format-namenode ']'
+ '[' transition-to-active = format-namenode ']'
+ '[' initializeSharedEdits = format-namenode ']'
+ '[' initialize-znode = format-namenode ']'
+ '[' format-namenode = format-namenode ']'
+ '[' -z /dfs/nn ']'
+ for dfsdir in '$DFS_STORAGE_DIRS'
+ '[' -e /dfs/nn ']'
+ '[' '!' -d /dfs/nn ']'
+ CLUSTER_ARGS=
+ '[' 2 -eq 2 ']'
+ CLUSTER_ARGS='-clusterId cluster19'
+ '[' 3 = 6 ']'
+ '[' -3 = 6 ']'
+ exec /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-hdfs/bin/hdfs --config /var/run/cloudera-scm-agent/process/67-hdfs-NAMENODE-format namenode -format -clusterId cluster19 -nonInteractive
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: HADOOP_NAMENODE_OPTS has been replaced by HDFS_NAMENODE_OPTS. Using value of HADOOP_NAMENODE_OPTS.
Running in non-interactive mode, and data appears to exist in Storage Directory /dfs/nn. Not formatting.

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

Super Guru

@rjotwani-211490215,

 

Please see my previous update.

 

"The supervisor_status file shouldn't need the config replacement at all so I opened Cloudera internal Jira OPSAPS-50270 to request we add it to the list of patterns that are excluded from replacing the placeholder in config files with the path to the process directory. That should address the issue so the Permission Denied message doesn't occur any more."

 

You can ignore the error as it has no impact on functionality.

If you are encountering a problem, please start a new community discussion and let us know the details so we can help out.

 

Cheers,

Ben

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

Explorer

Having the same issue with HUE now even though the database connection works fine:

 

 

==> cloudera-scm-server.log <==
2019-08-12 00:18:37,663 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HueTestDatabaseConnection'
2019-08-12 00:18:37,665 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 127
2019-08-12 00:18:37,665 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: + find /var/run/cloudera-scm-agent/process/69-HUE-test-db-connection -type f '!' -path '/var/run/cloudera-scm-agent/process/69-HUE-test-db-connection/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name creds.localjceks -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/69-HUE-test-db-connection#g' '{}' ';'
Can't open /var/run/cloudera-scm-agent/process/69-HUE-test-db-connection/supervisor_status: Permission denied.
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/69-HUE-test-db-connection -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ '[' is_db_alive == beeswax_server ']'
+ set_classpath_in_var HADOOP_EXTRA_CLASSPATH_STRING
+ '[' -z HADOOP_EXTRA_CLASSPATH_STRING ']'
+ [[ -n /opt/cloudera/cm ]]
++ find /opt/cloudera/cm/lib/plugins -maxdepth 1 -name '*.jar'
++ tr '\n' :
+ ADD_TO_CP=/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar:
+ [[ -n '' ]]
+ eval 'OLD_VALUE=$HADOOP_EXTRA_CLASSPATH_STRING'
++ OLD_VALUE=
+ NEW_VALUE=/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar:
+ export HADOOP_EXTRA_CLASSPATH_STRING=/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar
+ HADOOP_EXTRA_CLASSPATH_STRING=/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar
+ HUE=/usr/lib/hue/build/env/bin/hue
+ [[ is_db_alive == runcpserver ]]
+ [[ is_db_alive == kt_renewer ]]
+ '[' dumpdata = is_db_alive ']'
+ '[' syncdb = is_db_alive ']'
+ '[' ldaptest = is_db_alive ']'
+ exec /usr/lib/hue/build/env/bin/hue is_db_alive
/opt/cloudera/cm-agent/service/hue/hue.sh: line 293: /usr/lib/hue/build/env/bin/hue: No such file or directory

2019-08-12 00:18:37,665 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 1334(HueTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.

==> cmf-server-nio.log <==

==> cloudera-scm-server.log <==

 

 

I'll try to apply my solution but this is preventing cluster services installation:

Hue

 Unexpected error. Unable to verify database connection.
 
This causes a grayed out continue button, which makes it a real issue.  I do see a missing package on the host where I'm running CM however I've forgotten how I configured the second namenode where the package exists now thought I'm sure  I had installed all packages required on both nodes.    Or perhaps something else should have installed the required packages?   Can't remember exactly.  Manually installed:
 
hue-4.2.0+cdh6.2.0-967373.el7.x86_64
 
 

 

Dependencies Resolved

===========================================================================================================================
 Package                       Arch                Version                                 Repository                 Size
===========================================================================================================================
Installing:
 hue                           x86_64              4.2.0+cdh6.2.0-967373.el7               cloudera-cdh               78 M
Installing for dependencies:
 avro-libs                     noarch              1.8.2+cdh6.2.0-967373.el7               cloudera-cdh               41 M
 bigtop-jsvc                   x86_64              1.0.10+cdh6.2.0-967373.el7              cloudera-cdh               29 k
 hadoop                        x86_64              3.0.0+cdh6.2.0-967373.el7               cloudera-cdh              103 M
 hadoop-client                 x86_64              3.0.0+cdh6.2.0-967373.el7               cloudera-cdh               25 k
 hadoop-hdfs                   x86_64              3.0.0+cdh6.2.0-967373.el7               cloudera-cdh               44 M
 hadoop-mapreduce              x86_64              3.0.0+cdh6.2.0-967373.el7               cloudera-cdh               20 M
 hadoop-yarn                   x86_64              3.0.0+cdh6.2.0-967373.el7               cloudera-cdh               17 M
 hive                          noarch              2.1.1+cdh6.2.0-967373.el7               cloudera-cdh              182 M
 hive-jdbc                     noarch              2.1.1+cdh6.2.0-967373.el7               cloudera-cdh               42 M
 libtidy                       x86_64              5.4.0-1.el7                             epel                      174 k
 parquet                       noarch              1.9.0+cdh6.2.0-967373.el7               cloudera-cdh               80 M
 parquet-format                noarch              2.3.1+cdh6.2.0-967373.el7               cloudera-cdh              753 k
 sentry                        noarch              2.1.0+cdh6.2.0-967373.el7               cloudera-cdh              101 M
 solr                          noarch              7.4.0+cdh6.2.0-967373.el7               cloudera-cdh               64 M
 zookeeper                     x86_64              3.4.5+cdh6.2.0-967373.el7               cloudera-cdh              4.0 M

Transaction Summary
===========================================================================================================================
Install  1 Package (+15 Dependent packages)

 

 
Shouldn't Cloudera distribute the packages?  Just curious.  Or perhaps I forgot a step somewhere?

 

Cheers,
TK

 

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

Explorer

Spoke too soon.  Manual install didn't do the trick.  I'm actually uncertain which nodes the error message is coming from so unsure where to install Hue now.

 

Will pick it up tomorrow.

Cheers,
TK

Highlighted

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

Super Guru

@TCloud,

 

The error that shows the cause of the failure to test the database is:

 

/opt/cloudera/cm-agent/service/hue/hue.sh: line 293: /usr/lib/hue/build/env/bin/hue: No such file or directory

 

I am curious if you are using packages or parcels for Hue.  It appears you may be using packages in which case you do need to install the necessary Hue package on the host where you want to add teh Hue service.  If you use parcels, all of this is automated.

 

If you still need to use packages for CDH, then I think the error above should be solved by installing the Hue package.

 

Cloudera Manager does not distribute CDH packages; management of packages is done manually via apt-get, yum, etc..  If you choose to use "parcels" then CM manages the deployment of parcels to each host.

 

The error is occurring since the "hue.sh" start script expects the hue executable to be /usr/lib/hue/build/env/bin/hue, but that file does not exist.  If you were using parcels for this cluster, then the path would be to /"opt/cloudera/parcels/CDH..."

 

 

 

Re: CM / CDH 6.2: Can't ..../process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission de

Explorer

Switching to parcels worked.  I could have just deployed things manually with rpm's but parcels make this much easier and so I switched to them.

 

Posted a new issue however with Oozie.  (Separate thread)