Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Service did not start successfully

avatar
New Contributor

Hi all,

 

I have problems installing CM by following the Path A from the homepage. I went through the installation process of http://localhost:7180 and installed 3 nodes. Unfortunately, no service has started. When I tried to start them manually, I run into the following error:

 

Service did not start successfully; not all of the required roles started: only 0/4 roles started. Reasons : Service has only 0 Service Monitor roles running instead of minimum required 1.
Supervisor returned FATAL. Please check the role log file, stderr, or stdout.

Stderr:

Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mi 29. Mär 15:31:41 CEST 2017
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mi 29. Mär 15:31:53 CEST 2017
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mi 29. Mär 15:32:07 CEST 2017
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mi 29. Mär 15:32:21 CEST 2017
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x

 

Stdout:

Mi 29. Mär 15:31:41 CEST 2017
JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
Removing any leveldbjni library files left over from previous runs
Executing: /usr/lib/jvm/java-7-oracle-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mgmt_mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295_pid7460.hprof -XX:OnOutOfMemoryError=/usr/lib/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*: com.cloudera.cmon.firehose.Main --pipeline-type HOST_MONITORING --mgmt-home /usr/share/cmf
Mi 29. Mär 15:31:53 CEST 2017
JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
Removing any leveldbjni library files left over from previous runs
Executing: /usr/lib/jvm/java-7-oracle-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mgmt_mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295_pid7723.hprof -XX:OnOutOfMemoryError=/usr/lib/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*: com.cloudera.cmon.firehose.Main --pipeline-type HOST_MONITORING --mgmt-home /usr/share/cmf
Mi 29. Mär 15:32:07 CEST 2017
JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
Removing any leveldbjni library files left over from previous runs
Executing: /usr/lib/jvm/java-7-oracle-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mgmt_mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295_pid7923.hprof -XX:OnOutOfMemoryError=/usr/lib/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*: com.cloudera.cmon.firehose.Main --pipeline-type HOST_MONITORING --mgmt-home /usr/share/cmf
Mi 29. Mär 15:32:21 CEST 2017
JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
Removing any leveldbjni library files left over from previous runs
Executing: /usr/lib/jvm/java-7-oracle-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mgmt_mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295_pid8146.hprof -XX:OnOutOfMemoryError=/usr/lib/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/31-cloudera-mgmt-HOSTMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*: com.cloudera.cmon.firehose.Main --pipeline-type HOST_MONITORING --mgmt-home /usr/share/cmf

 

I cannot see a message which is helping. Can anybody help me?

 

Best regards,

Manuel

 

1 ACCEPTED SOLUTION

avatar
New Contributor

In case somebody is interessted in the solution: In my limited space on the partition causes the problems with the cloudera installation. That is, the message

 

/dev/sda1/
6.7 GiB/7.8 GiB

 

was a good hint to the solution.

 

Best,

Manuel

View solution in original post

3 REPLIES 3

avatar
Master Guru

Hello @ManuelAllh,

 

Sorry to hear you hit some problems installing and running.

The stderr.log and stdout.log do not show anything unsual, so we should take a look at the agent log on the host where you are trying to start the services.  The log is /var/log/cloudera-scm-agent/cloudera-scm-agent.log

 

You might tail -f that file while trying to start the services that are not starting.

If you don't see anything obvious that leads you to a solution, share the logging with us.

 

Also, it is possible there is a failure that is captured in the role log.  In your case, it appears the failure is during the start of the Host Monitor.  I recommend checking the Host Monitor log to see if there is anything useful there:

 

/var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out

 

The information in the agent and role log should help point us toward the cause.

 

-Ben

 

avatar
New Contributor

Hi Ben,

 

Many thanks for the prompt answer. There is a lot of stuff going on in the log files, but I cannot solve the problem.

 

My setup:

I have three PCs where I like to install claudera with IPs 192.168.50.101-103 (they are virtual instances on three different PCs). I can ping every PC from every PC. Due to our IT-company policies, only 192.168.50.3 has internet access, but I am nating the packages to the LAN, such that every PC has internet access.

 

I run the cloudera web-installer on 192.168.50.102, but the roles are assigned to 192.168.50.101. I guess that is normal. However, when I check the file system of 192.168.50.101 I get

 

/dev/sda1/
6.7 GiB/7.8 GiB

 

That seems a little tight for me. Could this cause problems?

 

The installer says there is no problem with /etc/hosts. However, this is the content of each of the three PCs (101-103)

 

127.0.0.1	localhost
#127.0.1.1	clbaseu1404

192.168.50.101	vengland
192.168.50.102	virland
192.168.50.103	vspanien

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

 

Tailing file /var/log/cloudera-scm-agent/cloudera-scm-agent.log of 101 gives

 

[30/Mar/2017 09:03:46 +0000] 17622 MainThread process      INFO     Deactivating process 46-cloudera-mgmt-EVENTSERVER
[30/Mar/2017 09:03:47 +0000] 17622 MonitorDaemon-Scheduler __init__     INFO     Monitor expired: ('GenericMonitor MGMT-EVENTSERVER for mgmt-EVENTSERVER-cf2e6beb6123f8a351ad9672d72ff295',)
[30/Mar/2017 09:03:47 +0000] 17622 MonitorDaemon-Scheduler __init__     INFO     Monitor expired: ('GenericMonitor MGMT-SERVICEMONITOR for mgmt-SERVICEMONITOR-cf2e6beb6123f8a351ad9672d72ff295',)
[30/Mar/2017 09:03:47 +0000] 17622 MonitorDaemon-Scheduler __init__     INFO     Monitor expired: ('GenericMonitor MGMT-HOSTMONITOR for mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295',)
[30/Mar/2017 09:03:47 +0000] 17622 MonitorDaemon-Scheduler __init__     INFO     Monitor expired: ('GenericMonitor MGMT-REPORTSMANAGER for mgmt-REPORTSMANAGER-cf2e6beb6123f8a351ad9672d72ff295',)
[30/Mar/2017 09:03:47 +0000] 17622 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for cloudera-mgmt-EVENTSERVER with pipelines []
[30/Mar/2017 09:03:47 +0000] 17622 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-EVENTSERVER (log dir: None)
[30/Mar/2017 09:03:47 +0000] 17622 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for cloudera-mgmt-EVENTSERVER with pipelines []
[30/Mar/2017 09:03:47 +0000] 17622 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-EVENTSERVER (log dir: None)
[30/Mar/2017 09:04:01 +0000] 17622 MainThread process      INFO     [43-cloudera-mgmt-REPORTSMANAGER] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
[30/Mar/2017 09:04:01 +0000] 17622 MainThread process      INFO     Deactivating process 43-cloudera-mgmt-REPORTSMANAGER
[30/Mar/2017 09:04:02 +0000] 17622 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for cloudera-mgmt-REPORTSMANAGER with pipelines []
[30/Mar/2017 09:04:02 +0000] 17622 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-REPORTSMANAGER (log dir: None)
[30/Mar/2017 09:04:02 +0000] 17622 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for cloudera-mgmt-REPORTSMANAGER with pipelines []
[30/Mar/2017 09:04:02 +0000] 17622 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-REPORTSMANAGER (log dir: None)
[30/Mar/2017 09:11:32 +0000] 17622 MainThread heartbeat_tracker INFO     HB stats (seconds): num:49 LIFE_MIN:0.03 min:0.01 mean:0.03 max:0.08 LIFE_MAX:0.07
[30/Mar/2017 09:12:28 +0000] 17622 MonitorDaemon-Reporter throttling_logger ERROR    (11 skipped) Error sending messages to firehose: mgmt-SERVICEMONITOR-cf2e6beb6123f8a351ad9672d72ff295
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err
error: [Errno 111] Connection refused
[30/Mar/2017 09:12:28 +0000] 17622 MonitorDaemon-Reporter throttling_logger ERROR    (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err
error: [Errno 111] Connection refused

Tailing file /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-vengland.log.out of 101 gives:

2017-03-30 09:03:28,935 INFO com.cloudera.cmon.TimeSeriesEntityType: Registered TimeSeriesEntityType HBASE_INDEXER
2017-03-30 09:03:28,942 INFO com.cloudera.cmon.firehose.CMONConfiguration: Config: file:/run/cloudera-scm-agent/process/42-cloudera-mgmt-HOSTMONITOR/cmon.conf
2017-03-30 09:03:29,112 INFO com.cloudera.cmf.cdhclient.util.CDHUrlClassLoader: Detected that this program is running in a JAVA 1.7.0_67 JVM. CDH5 jars will be loaded from:lib/cdh5
2017-03-30 09:03:29,113 INFO com.cloudera.enterprise.ssl.SSLFactory: Using default java truststore for verification of server certificates in HTTPS communication.
2017-03-30 09:03:29,219 WARN com.cloudera.cmf.BasicScmProxy: Exception while getting fetch configDefaults hash: none
java.net.UnknownHostException: localhost.localdomain
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
	at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:265)
	at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:561)
	at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:651)
	at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:64)
	at com.cloudera.cmon.firehose.Main.main(Main.java:378)
2017-03-30 09:03:29,223 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from http://localhost.localdomain:7180 on after 1 tries, sleeping...
2017-03-30 09:03:29,291 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={STACKTRACE=[java.net.UnknownHostException: localhost.localdomain
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
	at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:265)
	at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:561)
	at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:651)
	at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:64)
	at com.cloudera.cmon.firehose.Main.main(Main.java:378)
], EXCEPTION_TYPES=[java.net.UnknownHostException], ROLE=[mgmt-HOSTMONITOR-cf2e6beb6123f8a351ad9672d72ff295], SEVERITY=[IMPORTANT], SERVICE=[mgmt], HOST_IDS=[dbdb858b-4c99-4122-9eef-62e4127b513d], LOG_LEVEL=[WARN], ROLE_TYPE=[HOSTMONITOR], CATEGORY=[LOG_MESSAGE], SERVICE_TYPE=[MGMT], HOSTS=[vengland], EVENTCODE=[EV_LOG_EVENT]}, content=Exception while getting fetch configDefaults hash: none, timestamp=1490857409219}
2017-03-30 09:03:31,242 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from http://localhost.localdomain:7180 on after 2 tries, sleeping...
2017-03-30 09:03:33,243 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from http://localhost.localdomain:7180 on after 3 tries, sleeping...
2017-03-30 09:03:35,245 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from http://localhost.localdomain:7180 on after 4 tries, sleeping...
2017-03-30 09:03:37,247 WARN com.cloudera.cmon.firehose.Main: No descriptor fetched from http://localhost.localdomain:7180 on after 5 tries, sleeping...
2017-03-30 09:03:39,249 ERROR com.cloudera.cmon.firehose.Main: Could not fetch descriptor after 5 tries, exiting.

It does not seem appropriate that 101 is trying to fetch something from localhost. Is that correct?

 

Any help is appreciated. Thanks in advance!

 

Best,

Manuel

 

avatar
New Contributor

In case somebody is interessted in the solution: In my limited space on the partition causes the problems with the cloudera installation. That is, the message

 

/dev/sda1/
6.7 GiB/7.8 GiB

 

was a good hint to the solution.

 

Best,

Manuel