Event Server fails to start after deleting some log files

[#1]

Please help me, I found log files making full on path after I deleted some logs in /var/log/cloudera-scm-agent and back to CM. I try to restart Cloudera Management Service and then I face Event Server cannot start anymore

My stdout err message show as below : 

[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:19 +0000] 29231 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29239
+ date
+ date
Fri Jan 14 19:41:20 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
[14/Jan/2022 19:41:23 +0000] 29298 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:23 +0000] 29298 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:23 +0000] 29298 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29305
+ date
+ date
Fri Jan 14 19:41:24 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
[14/Jan/2022 19:41:28 +0000] 29365 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:28 +0000] 29365 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:28 +0000] 29365 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29372
+ date
+ date
Fri Jan 14 19:41:29 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
[14/Jan/2022 19:41:35 +0000] 29453 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/mgmt/mgmt.sh eventserver
[14/Jan/2022 19:41:35 +0000] 29453 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
[14/Jan/2022 19:41:35 +0000] 29453 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 29461
+ date
+ date
Fri Jan 14 19:41:35 +07 2022
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /opt/cloudera/cm-agent/service/mgmt/mgmt.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/mgmt
++ cd /opt/cloudera/cm-agent/service/mgmt/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]})
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_221-amd64 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER#g' '{}' ';'
Can't open /var/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied.
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/java/jdk1.8.0_221-amd64 '!=' '' ']'
+ JAVA=/usr/java/jdk1.8.0_221-amd64/bin/java
+ [[ -n /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ [[ -z /opt/cloudera/cm/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ DDL_DIR=/opt/cloudera/cm/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x

[#2 · Accepted solution]

The data directory for the event server may be corrupt. You can try the following:

1. Make sure the Event server role is stopped.

2. Edit the current data directory (default is /var/lib/cloudera-scm-eventserver), by adding a -2' or 'old',etc.

3. Start the Event server.

If it starts , then it is a good indicator that the data directory became corrupt.

[#3]

 Does the "/run/cloudera-scm-agent/process/22986-cloudera-mgmt-EVENTSERVER" directory exist?

[#4]

Yes, It's there and the all files & sub folder are inside here. I expected that some event folder in path : /run/cloudera-scm-agent/process cannot be marked up to the logs files. Some log file, I may have accidentally deleted it.

[#5 · Accepted solution]

The data directory for the event server may be corrupt. You can try the following:

1. Make sure the Event server role is stopped.

2. Edit the current data directory (default is /var/lib/cloudera-scm-eventserver), by adding a -2' or 'old',etc.

3. Start the Event server.

If it starts , then it is a good indicator that the data directory became corrupt.

[#6 · Accepted solution]

Hi Elias,

Your workaround is so perfectly, I try it one time and it's successfully.  Wow! I need to keep your knowledge to be reference next.

Thank you very much for your helping and have a great day!