Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Event Server fails to start after deleting some log files

New Contributor

Please help me, I found log files making full on path after I deleted some logs in /var/log/cloudera-scm-agent and back to CM. I try to restart Cloudera Management Service and then I face Event Server cannot start anymore

 

My stdout err message show as below : 

[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29239
+ date
+ date
Fri Jan 14 19:41:20 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
[14/Jan/2022 19:41:23 +0000] 29298 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:23 +0000] 29298 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:23 +0000] 29298 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29305
+ date
+ date
Fri Jan 14 19:41:24 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
[14/Jan/2022 19:41:28 +0000] 29365 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:28 +0000] 29365 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:28 +0000] 29365 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29372
+ date
+ date
Fri Jan 14 19:41:29 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
[14/Jan/2022 19:41:35 +0000] 29453 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:35 +0000] 29453 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:35 +0000] 29453 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29461
+ date
+ date
Fri Jan 14 19:41:35 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
Can't open /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied.
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x

 

Capture1.PNG
Capture2.PNG
Capture3.PNG
2 ACCEPTED SOLUTIONS

Cloudera Employee

The data directory for the event server may be corrupt. You can try the following:

1. Make sure the Event server role is stopped.

2. Edit the current data directory (default is /var/lib/cloudera-scm-eventserver), by adding a -2' or 'old',etc.

3. Start the Event server.

If it starts , then it is a good indicator that the data directory became corrupt.

View solution in original post

New Contributor

Hi Elias,

 

Your workaround is so perfectly, I try it one time and it's successfully.  Wow! I need to keep your knowledge to be reference next.

 

Thank you very much for your helping and have a great day! 

View solution in original post

EventServer_back_19Jan2022.PNG
4 REPLIES 4

Cloudera Employee

 Does the "/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER" directory exist?

 

New Contributor

Yes, It's there and the all files & sub folder are inside here. I expected that some event folder in path : /run/cloudera-scm-agent/process cannot be marked up to the logs files. Some log file, I may have accidentally deleted it.

Capture15.PNG

Cloudera Employee

The data directory for the event server may be corrupt. You can try the following:

1. Make sure the Event server role is stopped.

2. Edit the current data directory (default is /var/lib/cloudera-scm-eventserver), by adding a -2' or 'old',etc.

3. Start the Event server.

If it starts , then it is a good indicator that the data directory became corrupt.

New Contributor

Hi Elias,

 

Your workaround is so perfectly, I try it one time and it's successfully.  Wow! I need to keep your knowledge to be reference next.

 

Thank you very much for your helping and have a great day! 

EventServer_back_19Jan2022.PNG