Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Host Monitor Not Starting

Host Monitor Not Starting

New Contributor

MY Host Monitor isnt starting after Cloudera Installation. Please help!

 

 

ERROR	com.cloudera.cmon.firehose.Main

Error creating LevelDB subject record store. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:133) at com.cloudera.cmon.tstore.leveldb.LDBResourceManager.register(LDBResourceManager.java:274) at com.cloudera.cmon.tstore.leveldb.LDBPartitionManager.<init>(LDBPartitionManager.java:270) at com.cloudera.cmon.tstore.leveldb.LDBPartitionManager.createLDBPartitionManager(LDBPartitionManager.java:196) at com.cloudera.cmon.tstore.leveldb.LDBTSSubjectRecordTable.create(LDBTSSubjectRecordTable.java:84) at com.cloudera.cmon.tstore.leveldb.LDBSubjectRecordStore.create(LDBSubjectRecordStore.java:197) at com.cloudera.cmon.tstore.leveldb.LDBSubjectRecordStore.create(LDBSubjectRecordStore.java:178) at com.cloudera.cmon.firehose.Main.main(Main.java:505)

 

 

4 REPLIES 4
Highlighted

Re: Host Monitor Not Starting

Super Collaborator

> MY Host Monitor isnt starting after Cloudera Installation. Please help!

 

Is this a new installation or is it an upgrade from previous version?

 

 

Can you confirm your current Cloudera Manager version, and also provide the output of the following commands:

 

 

[root@host-monitor]# grep -C 5 HOSTMONITOR /var/log/cloudera-scm-agent/supervisord.log

[root@host-monitor]# grep "com.cloudera.cmon.firehose.Main: Starting Firehose." /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR*.out | tail -1

 

Other things to check, are the Agent logs and the HOSTMONITOR logs

 

/var/log/cloudera-scm-agent/cloudera-scm-agent.log

/var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-*.log.out

 

 

-

Michalis

 

Re: Host Monitor Not Starting

New Contributor

Its a new Installation of

Cloudera Express 5.5.1

 

 

myname@mycomputer:~$ grep "com.cloudera.cmon.firehose.Main: Starting Firehose." /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR*.out | tail -1
2016-01-21 09:49:59,869 INFO com.cloudera.cmon.firehose.Main: Starting Firehose. JVM Args: [-XX:+UseConcMarkSweepGC, -XX:+UseParNewGC, -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-big03-itrc.bmwgroup.net.log.out, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Dfirehose.schema.dir=/usr/share/cmf/schema, -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/73-cloudera-mgmt-HOSTMONITOR, -Xms1073741824, -Xmx1073741824, -XX:OnOutOfMemoryError=/usr/lib/cmf/service/common/killparent.sh], Args: [--pipeline-type, HOST_MONITORING, --mgmt-home, /usr/share/cmf], Version: 5.5.1 (#8 built by jenkins on 20151201-1818 git: 2a7dfe22d921bef89c7ee3c2981cb4c1dc43de7b)

 

2016-01-21 09:43:40,680 INFO exited: 68-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:43:41,732 INFO spawned: '68-cloudera-mgmt-HOSTMONITOR' with pid 4167
2016-01-21 09:43:41,732 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:43:41,733 INFO exited: 71-cloudera-mgmt-SERVICEMONITOR (exit status 1; not expected)
2016-01-21 09:43:42,765 INFO spawned: '71-cloudera-mgmt-SERVICEMONITOR' with pid 4215
2016-01-21 09:43:42,765 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:43:48,144 INFO exited: 68-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:43:49,784 INFO exited: 71-cloudera-mgmt-SERVICEMONITOR (exit status 1; not expected)
2016-01-21 09:43:50,787 INFO spawned: '68-cloudera-mgmt-HOSTMONITOR' with pid 4433
2016-01-21 09:43:50,788 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:43:51,822 INFO spawned: '71-cloudera-mgmt-SERVICEMONITOR' with pid 4480
2016-01-21 09:43:51,823 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:43:53,851 INFO success: 67-cloudera-mgmt-ALERTPUBLISHER entered RUNNING state, process has stayed up for > than 20 seconds (startsecs)
2016-01-21 09:43:54,852 INFO success: 70-cloudera-mgmt-ACTIVITYMONITOR entered RUNNING state, process has stayed up for > than 20 seconds (startsecs)
2016-01-21 09:43:54,852 INFO success: 69-cloudera-mgmt-EVENTSERVER entered RUNNING state, process has stayed up for > than 20 seconds (startsecs)
2016-01-21 09:43:56,944 INFO exited: 68-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:43:58,051 INFO exited: 71-cloudera-mgmt-SERVICEMONITOR (exit status 1; not expected)
2016-01-21 09:44:00,056 INFO spawned: '68-cloudera-mgmt-HOSTMONITOR' with pid 4721
2016-01-21 09:44:00,056 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:44:01,091 INFO spawned: '71-cloudera-mgmt-SERVICEMONITOR' with pid 4768
2016-01-21 09:44:01,091 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:44:06,274 INFO exited: 68-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:44:07,276 INFO gave up: 68-cloudera-mgmt-HOSTMONITOR entered FATAL state, too many start retries too quickly
2016-01-21 09:44:07,327 INFO exited: 71-cloudera-mgmt-SERVICEMONITOR (exit status 1; not expected)
2016-01-21 09:44:07,327 INFO gave up: 71-cloudera-mgmt-SERVICEMONITOR entered FATAL state, too many start retries too quickly
2016-01-21 09:48:12,795 INFO spawned: '72-cloudera-mgmt-HOSTMONITOR' with pid 6881
2016-01-21 09:48:12,795 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:48:18,919 INFO exited: 72-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:48:19,923 INFO spawned: '72-cloudera-mgmt-HOSTMONITOR' with pid 6939
2016-01-21 09:48:19,923 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:48:27,097 INFO exited: 72-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:48:29,103 INFO spawned: '72-cloudera-mgmt-HOSTMONITOR' with pid 7085
2016-01-21 09:48:29,103 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:48:35,700 INFO exited: 72-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:48:38,707 INFO spawned: '72-cloudera-mgmt-HOSTMONITOR' with pid 7171
2016-01-21 09:48:38,707 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:48:45,649 INFO exited: 72-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:48:46,651 INFO gave up: 72-cloudera-mgmt-HOSTMONITOR entered FATAL state, too many start retries too quickly
2016-01-21 09:49:32,303 INFO spawned: '73-cloudera-mgmt-HOSTMONITOR' with pid 7709
2016-01-21 09:49:32,303 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:49:38,499 INFO exited: 73-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:49:39,502 INFO spawned: '73-cloudera-mgmt-HOSTMONITOR' with pid 7796
2016-01-21 09:49:39,503 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:49:46,168 INFO exited: 73-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:49:48,616 INFO spawned: '73-cloudera-mgmt-HOSTMONITOR' with pid 7943
2016-01-21 09:49:48,617 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:49:56,207 INFO exited: 73-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:49:59,213 INFO spawned: '73-cloudera-mgmt-HOSTMONITOR' with pid 7999
2016-01-21 09:49:59,214 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:50:06,669 INFO exited: 73-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:50:07,670 INFO gave up: 73-cloudera-mgmt-HOSTMONITOR entered FATAL state, too many start retries too quickly

Re: Host Monitor Not Starting

Super Collaborator

If this is on Ubuntu, there's a know bug on that OS, can you set the ulimit to 32k (~32768) and restart HMON

 

Navigate to:

CM> MGMT> Host Monitor Default Group / Performance> Maximum Process File Descriptors
CM> MGMT> Service Monitor Default Group / Performance> Maximum Process File Descriptors

Re: Host Monitor Not Starting

New Contributor

THanks but I Re-installed CDH and it fixed the issue.