Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Event Server fails to start after deleting some log files

avatar
Explorer

Please help me, I found log files making full on path after I deleted some logs in /var/log/cloudera-scm-agent and back to CM. I try to restart Cloudera Management Service and then I face Event Server cannot start anymore

 

My stdout err message show as below : 

[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29239
+ date
+ date
Fri Jan 14 19:41:20 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
[14/Jan/2022 19:41:23 +0000] 29298 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:23 +0000] 29298 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:23 +0000] 29298 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29305
+ date
+ date
Fri Jan 14 19:41:24 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
[14/Jan/2022 19:41:28 +0000] 29365 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:28 +0000] 29365 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:28 +0000] 29365 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29372
+ date
+ date
Fri Jan 14 19:41:29 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
[14/Jan/2022 19:41:35 +0000] 29453 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:35 +0000] 29453 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:35 +0000] 29453 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29461
+ date
+ date
Fri Jan 14 19:41:35 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
Can't open /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied.
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x

 


Capture1.PNGCapture2.PNGCapture3.PNG
2 ACCEPTED SOLUTIONS

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Expert Contributor

 Does the "/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER" directory exist?

 

avatar
Explorer

Yes, It's there and the all files & sub folder are inside here. I expected that some event folder in path : /run/cloudera-scm-agent/process cannot be marked up to the logs files. Some log file, I may have accidentally deleted it.


Capture15.PNG

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login