Reply
Max
New Contributor
Posts: 2
Registered: ‎01-08-2018

Host Monitor not running, fresh install

[ Edited ]

Hello,

 

I'm trying to deploy a test CDH 5.13.1 on a cluster of 4 AWS machines with  60Gb of RAM each (Ubuntu 16.04)

 

The deploy/discovery is successful but the dashboard complains Host Monitor is not running (already at teh cluster installation phase, with "host bad health" msg), hence it is missing completely the info about the cluster status.

 

 

I saw 2 warnings, complaining about non-Java memeory limit being lower than the recommended value (out of the box?), I think I increased it in accord using the dashboard, as after rebooting all the hosts I don't see the warnings anymore...how can I verify?

 

here the stderr from when I tried to start the host monitor service:

 

/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration
  global REDACTOR
/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration
  global NOP_REDACTOR
Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mon Jan  8 16:06:19 UTC 2018
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ replace_conf_dir
+ echo CONF_DIR=/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR
+ echo CMF_CONF_DIR=/etc/cloudera-scm-agent
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'httpfs.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh httpfs.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR -type f '!' -path '/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR/logs/*' '!' -name cloudera-config.sh '!' -name httpfs.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR#g' '{}' ';'
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-spot3.extreme.com.log.out (No such file or directory)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
	at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
	at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
	at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
	at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
	at com.cloudera.enterprise.CommonMain.<clinit>(CommonMain.java:51)
	at com.cloudera.cmon.firehose.Main.<clinit>(Main.java:53)
/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration
  global REDACTOR
/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration
  global NOP_REDACTOR
Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mon Jan  8 16:06:26 UTC 2018
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ replace_conf_dir
+ echo CONF_DIR=/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR
+ echo CMF_CONF_DIR=/etc/cloudera-scm-agent
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'httpfs.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh httpfs.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR -type f '!' -path '/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR/logs/*' '!' -name cloudera-config.sh '!' -name httpfs.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR#g' '{}' ';'
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-spot3.extreme.com.log.out (No such file or directory)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
	at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
	at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
	at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
	at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
	at com.cloudera.enterprise.CommonMain.<clinit>(CommonMain.java:51)
	at com.cloudera.cmon.firehose.Main.<clinit>(Main.java:53)
/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration
  global REDACTOR
/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration
  global NOP_REDACTOR
Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mon Jan  8 16:06:34 UTC 2018
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ replace_conf_dir
+ echo CONF_DIR=/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR
+ echo CMF_CONF_DIR=/etc/cloudera-scm-agent
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'httpfs.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh httpfs.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR -type f '!' -path '/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR/logs/*' '!' -name cloudera-config.sh '!' -name httpfs.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR#g' '{}' ';'
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-spot3.extreme.com.log.out (No such file or directory)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
	at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
	at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
	at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
	at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
	at com.cloudera.enterprise.CommonMain.<clinit>(CommonMain.java:51)
	at com.cloudera.cmon.firehose.Main.<clinit>(Main.java:53)
/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration
  global REDACTOR
/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration
  global NOP_REDACTOR
Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR
+ date
+ date
Mon Jan  8 16:06:43 UTC 2018
+ umask 027
+ BASE_JAVA_OPTS='-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC'
+ MGMT_HOME_ARG_NAME=--mgmt-home
++ dirname /usr/lib/cmf/service/mgmt/mgmt.sh
+ cloudera_config=/usr/lib/cmf/service/mgmt
++ cd /usr/lib/cmf/service/mgmt/../common
++ pwd
+ cloudera_config=/usr/lib/cmf/service/common
+ . /usr/lib/cmf/service/common/cloudera-config.sh
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jdk1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/java/jre1.7*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd '/usr/lib/jvm/j2sdk1.7-oracle/jre*'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/lib/jvm/java-7-oracle-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/lib/jvm/java-7-oracle-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/lib/jvm/java-7-oracle-cloudera ']'
+ echo JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
+ replace_conf_dir
+ echo CONF_DIR=/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR
+ echo CMF_CONF_DIR=/etc/cloudera-scm-agent
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'httpfs.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks')
++ printf '! -name %s ' cloudera-config.sh httpfs.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks'
+ find /run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR -type f '!' -path '/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR/logs/*' '!' -name cloudera-config.sh '!' -name httpfs.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR#g' '{}' ';'
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/{{ <-- HERE CMF_CONF_DIR}}/ at -e line 1.
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ '[' /usr/lib/jvm/java-7-oracle-cloudera '!=' '' ']'
+ JAVA=/usr/lib/jvm/java-7-oracle-cloudera/bin/java
+ [[ -n /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ [[ -z /usr/share/cmf/lib/* ]]
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*:'
+ DDL_DIR=/usr/share/cmf/schema
+ [[ false == \f\a\l\s\e ]]
+ set +x
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-spot3.extreme.com.log.out (No such file or directory)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
	at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
	at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
	at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
	at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
	at com.cloudera.enterprise.CommonMain.<clinit>(CommonMain.java:51)
	at com.cloudera.cmon.firehose.Main.<clinit>(Main.java:53)

Any help is really appreciated, thanks!

 

Max

Posts: 1,045
Topics: 1
Kudos: 263
Solutions: 131
Registered: ‎04-22-2014

Re: Host Monitor not running, fresh install

@Max,

 

It appears that there is a problem writing to the Host Monitor log file:

 

java.io.FileNotFoundException: /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-spot3.extreme.com.log.out (No such file or directory)
	at java.io.FileOutputStream.open(Native Method)

 

This is unusual and may be related to the problem.  Can you check to see if there is a /var/log/cloudera-scm-firehose directory and, if so, what are the permissions/ownership on that directory?

 

Also, what files are in "/var/log/cloudera-scm-firehose/"?

 

 

Max
New Contributor
Posts: 2
Registered: ‎01-08-2018

Re: Host Monitor not running, fresh install

Hello, thanks for your reply, I had to retry the deploy procedure (uninstalled/installed the manager) and now I'm having a problem as the nodes are already registered despite every attempt to decommissioning (quite frustrating to be honest)

 

I'll try to get back to the point I was and will update the thread, thanks.

 

Max

Announcements