question Re: Service did not start successfully in Archives of Support Questions (Read Only)

Service did not start successfully

ManuelAllh — Wed, 29 Mar 2017 13:55:03 GMT

Hi all,

I have problems installing CM by following the Path A from the homepage. I went through the installation process of http://localhost:7180 and installed 3 nodes. Unfortunately, no service has started. When I tried to start them manually, I run into the following error:

Service did not start successfully; not all of the required roles started: only 0/4 roles started. Reasons : Service has only 0 Service Monitor roles running instead of minimum required 1.

Supervisor returned FATAL. Please check the role log file, stderr, or stdout.

Stderr:

Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mi 29. Mär 15:31:41 CEST 2017
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mi 29. Mär 15:31:53 CEST 2017
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mi 29. Mär 15:32:07 CEST 2017
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mi 29. Mär 15:32:21 CEST 2017
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x

Stdout:

Mi 29. Mär 15:31:41 CEST 2017
JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
Removing any leveldbjni library files left over from previous runs
Executing: /usr/lib/jvm/java-7-oracle-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mgmt_mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295_pid7460.hprof -XX:OnOutOfMemoryError=/usr/lib/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*: com.cloudera.cmon.firehose.Main --pipeline-type HOST_MONITORING --mgmt-home /usr/share/cmf
Mi 29. Mär 15:31:53 CEST 2017
JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
Removing any leveldbjni library files left over from previous runs
Executing: /usr/lib/jvm/java-7-oracle-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mgmt_mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295_pid7723.hprof -XX:OnOutOfMemoryError=/usr/lib/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*: com.cloudera.cmon.firehose.Main --pipeline-type HOST_MONITORING --mgmt-home /usr/share/cmf
Mi 29. Mär 15:32:07 CEST 2017
JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
Removing any leveldbjni library files left over from previous runs
Executing: /usr/lib/jvm/java-7-oracle-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mgmt_mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295_pid7923.hprof -XX:OnOutOfMemoryError=/usr/lib/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*: com.cloudera.cmon.firehose.Main --pipeline-type HOST_MONITORING --mgmt-home /usr/share/cmf
Mi 29. Mär 15:32:21 CEST 2017
JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
Removing any leveldbjni library files left over from previous runs
Executing: /usr/lib/jvm/java-7-oracle-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mgmt_mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295_pid8146.hprof -XX:OnOutOfMemoryError=/usr/lib/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*: com.cloudera.cmon.firehose.Main --pipeline-type HOST_MONITORING --mgmt-home /usr/share/cmf

I cannot see a message which is helping. Can anybody help me?

Best regards,

Manuel

Re: Service did not start successfully

bgooley — Wed, 29 Mar 2017 14:18:21 GMT

Hello @ManuelAllh,

Sorry to hear you hit some problems installing and running.

The stderr.log and stdout.log do not show anything unsual, so we should take a look at the agent log on the host where you are trying to start the services. The log is /var/log/cloudera-scm-agent/cloudera-scm-agent.log

You might tail -f that file while trying to start the services that are not starting.

If you don't see anything obvious that leads you to a solution, share the logging with us.

Also, it is possible there is a failure that is captured in the role log. In your case, it appears the failure is during the start of the Host Monitor. I recommend checking the Host Monitor log to see if there is anything useful there:

/var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out

The information in the agent and role log should help point us toward the cause.

-Ben

Re: Service did not start successfully

ManuelAllh — Thu, 30 Mar 2017 07:22:26 GMT

Hi Ben,

Many thanks for the prompt answer. There is a lot of stuff going on in the log files, but I cannot solve the problem.

My setup:

I have three PCs where I like to install claudera with IPs 192.168.50.101-103 (they are virtual instances on three different PCs). I can ping every PC from every PC. Due to our IT-company policies, only 192.168.50.3 has internet access, but I am nating the packages to the LAN, such that every PC has internet access.

I run the cloudera web-installer on 192.168.50.102, but the roles are assigned to 192.168.50.101. I guess that is normal. However, when I check the file system of 192.168.50.101 I get

/dev/sda1

6.7 GiB/7.8 GiB

That seems a little tight for me. Could this cause problems?

The installer says there is no problem with /etc/hosts. However, this is the content of each of the three PCs (101-103)

127.0.0.1	localhost
#127.0.1.1	clbaseu1404

192.168.50.101	vengland
192.168.50.102	virland
192.168.50.103	vspanien

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Tailing file /var/log/cloudera-scm-agent/cloudera-scm-agent.log of 101 gives

[30/Mar/2017 09:03:46 +0000] 17622 MainThread process      INFO     Deactivating process 46-cloudera-mgmt-EVENTSERVER
[30/Mar/2017 09:03:47 +0000] 17622 MonitorDaemon-Scheduler __init__     INFO     Monitor expired: ('GenericMonitor MGMT-EVENTSERVER for mgmt-EVENTSERVER-cf2e6beb6123f8a351ad9672d72ff295',)
[30/Mar/2017 09:03:47 +0000] 17622 MonitorDaemon-Scheduler __init__     INFO     Monitor expired: ('GenericMonitor MGMT-SERVICEMONITOR for mgmt-SERVICEMONITOR-cf2e6beb6123f8a351ad9672d72ff295',)
[30/Mar/2017 09:03:47 +0000] 17622 MonitorDaemon-Scheduler __init__     INFO     Monitor expired: ('GenericMonitor MGMT-HOSTMONITOR for mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295',)
[30/Mar/2017 09:03:47 +0000] 17622 MonitorDaemon-Scheduler __init__     INFO     Monitor expired: ('GenericMonitor MGMT-REPORTSMANAGER for mgmt-REPORTSMANAGER-cf2e6beb6123f8a351ad9672d72ff295',)
[30/Mar/2017 09:03:47 +0000] 17622 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for cloudera-mgmt-EVENTSERVER with pipelines []
[30/Mar/2017 09:03:47 +0000] 17622 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-EVENTSERVER (log dir: None)
[30/Mar/2017 09:03:47 +0000] 17622 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for cloudera-mgmt-EVENTSERVER with pipelines []
[30/Mar/2017 09:03:47 +0000] 17622 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-EVENTSERVER (log dir: None)
[30/Mar/2017 09:04:01 +0000] 17622 MainThread process      INFO     [43-cloudera-mgmt-REPORTSMANAGER] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
[30/Mar/2017 09:04:01 +0000] 17622 MainThread process      INFO     Deactivating process 43-cloudera-mgmt-REPORTSMANAGER
[30/Mar/2017 09:04:02 +0000] 17622 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for cloudera-mgmt-REPORTSMANAGER with pipelines []
[30/Mar/2017 09:04:02 +0000] 17622 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-REPORTSMANAGER (log dir: None)
[30/Mar/2017 09:04:02 +0000] 17622 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for cloudera-mgmt-REPORTSMANAGER with pipelines []
[30/Mar/2017 09:04:02 +0000] 17622 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-REPORTSMANAGER (log dir: None)
[30/Mar/2017 09:11:32 +0000] 17622 MainThread heartbeat_tracker INFO     HB stats (seconds): num:49 LIFE_MIN:0.03 min:0.01 mean:0.03 max:0.08 LIFE_MAX:0.07
[30/Mar/2017 09:12:28 +0000] 17622 MonitorDaemon-Reporter throttling_logger ERROR    (11 skipped) Error sending messages to firehose: mgmt-SERVICEMONITOR-cf2e6beb6123f8a351ad9672d72ff295
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err
error: [Errno 111] Connection refused
[30/Mar/2017 09:12:28 +0000] 17622 MonitorDaemon-Reporter throttling_logger ERROR    (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err
error: [Errno 111] Connection refused

Tailing file /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out of 101 gives:

2017-03-30 09:03:28,935 INFO com.cloudera.cmon.TimeSeriesEntityType: Registered TimeSeriesEntityType HBASE_INDEXER
2017-03-30 09:03:28,942 INFO com.cloudera.cmon.firehose.CMONConfiguration: Config: file:/run/cloudera-scm-agent/process/42-cloudera-mgmt-HOSTMONITOR/cmon.conf
2017-03-30 09:03:29,112 INFO com.cloudera.cmf.cdhclient.util.CDHUrlClassLoader: Detected that this program is running in a JAVA 1.7.0_67 JVM. CDH5 jars will be loaded from:lib/cdh5
2017-03-30 09:03:29,113 INFO com.cloudera.enterprise.ssl.SSLFactory: Using default java truststore for verification of server certificates in HTTPS communication.
2017-03-30 09:03:29,219 WARN com.cloudera.cmf.BasicScmProxy: Exception while getting fetch configDefaults hash: none
java.net.UnknownHostException: localhost.localdomain
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
	at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:265)
	at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:561)
	at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:651)
	at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:64)
	at com.cloudera.cmon.firehose.Main.main(Main.java:378)
2017-03-30 09:03:29,223 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from http://localhost.localdomain:7180 on after 1 tries, sleeping...
2017-03-30 09:03:29,291 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={STACKTRACE=[java.net.UnknownHostException: localhost.localdomain
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
	at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:265)
	at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:561)
	at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:651)
	at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:64)
	at com.cloudera.cmon.firehose.Main.main(Main.java:378)
], EXCEPTION_TYPES=[java.net.UnknownHostException], ROLE=[mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295], SEVERITY=[IMPORTANT], SERVICE=[mgmt], HOST_IDS=[dbdb858b-4c99-4122-9eef-62e4127b513d], LOG_LEVEL=[WARN], ROLE_TYPE=[HOSTMONITOR], CATEGORY=[LOG_MESSAGE], SERVICE_TYPE=[MGMT], HOSTS=[vengland], EVENTCODE=[EV_LOG_EVENT]}, content=Exception while getting fetch configDefaults hash: none, timestamp=1490857409219}
2017-03-30 09:03:31,242 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from http://localhost.localdomain:7180 on after 2 tries, sleeping...
2017-03-30 09:03:33,243 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from http://localhost.localdomain:7180 on after 3 tries, sleeping...
2017-03-30 09:03:35,245 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from http://localhost.localdomain:7180 on after 4 tries, sleeping...
2017-03-30 09:03:37,247 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from http://localhost.localdomain:7180 on after 5 tries, sleeping...
2017-03-30 09:03:39,249 ERROR com.cloudera.cmon.firehose.Main: Could not fetch descriptor after 5 tries, exiting.

It does not seem appropriate that 101 is trying to fetch something from localhost. Is that correct?

Any help is appreciated. Thanks in advance!

Best,

Manuel

Re: Service did not start successfully

ManuelAllh — Thu, 06 Apr 2017 09:30:55 GMT

In case somebody is interessted in the solution: In my limited space on the partition causes the problems with the cloudera installation. That is, the message

/dev/sda1

6.7 GiB/7.8 GiB

was a good hint to the solution.

Best,

Manuel