Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎01-21-2016

Host Monitor Not Starting

MY Host Monitor isnt starting after Cloudera Installation. Please help!

 

 

ERROR	com.cloudera.cmon.firehose.Main

Error creating LevelDB subject record store. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:133) at com.cloudera.cmon.tstore.leveldb.LDBResourceManager.register(LDBResourceManager.java:274) at com.cloudera.cmon.tstore.leveldb.LDBPartitionManager.<init>(LDBPartitionManager.java:270) at com.cloudera.cmon.tstore.leveldb.LDBPartitionManager.createLDBPartitionManager(LDBPartitionManager.java:196) at com.cloudera.cmon.tstore.leveldb.LDBTSSubjectRecordTable.create(LDBTSSubjectRecordTable.java:84) at com.cloudera.cmon.tstore.leveldb.LDBSubjectRecordStore.create(LDBSubjectRecordStore.java:197) at com.cloudera.cmon.tstore.leveldb.LDBSubjectRecordStore.create(LDBSubjectRecordStore.java:178) at com.cloudera.cmon.firehose.Main.main(Main.java:505)

 

 

Cloudera Employee
Posts: 289
Registered: ‎07-08-2013

Re: Host Monitor Not Starting

> MY Host Monitor isnt starting after Cloudera Installation. Please help!

 

Is this a new installation or is it an upgrade from previous version?

 

 

Can you confirm your current Cloudera Manager version, and also provide the output of the following commands:

 

 

[root@host-monitor]# grep -C 5 HOSTMONITOR /var/log/cloudera-scm-agent/supervisord.log

[root@host-monitor]# grep "com.cloudera.cmon.firehose.Main: Starting Firehose." /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR*.out | tail -1

 

Other things to check, are the Agent logs and the HOSTMONITOR logs

 

/var/log/cloudera-scm-agent/cloudera-scm-agent.log

/var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-*.log.out

 

 

-

Michalis

 

New Contributor
Posts: 3
Registered: ‎01-21-2016

Re: Host Monitor Not Starting

Its a new Installation of

Cloudera Express 5.5.1

 

 

myname@mycomputer:~$ grep "com.cloudera.cmon.firehose.Main: Starting Firehose." /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR*.out | tail -1
2016-01-21 09:49:59,869 INFO com.cloudera.cmon.firehose.Main: Starting Firehose. JVM Args: [-XX:+UseConcMarkSweepGC, -XX:+UseParNewGC, -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-big03-itrc.bmwgroup.net.log.out, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Dfirehose.schema.dir=/usr/share/cmf/schema, -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/73-cloudera-mgmt-HOSTMONITOR, -Xms1073741824, -Xmx1073741824, -XX:OnOutOfMemoryError=/usr/lib/cmf/service/common/killparent.sh], Args: [--pipeline-type, HOST_MONITORING, --mgmt-home, /usr/share/cmf], Version: 5.5.1 (#8 built by jenkins on 20151201-1818 git: 2a7dfe22d921bef89c7ee3c2981cb4c1dc43de7b)

 

2016-01-21 09:43:40,680 INFO exited: 68-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:43:41,732 INFO spawned: '68-cloudera-mgmt-HOSTMONITOR' with pid 4167
2016-01-21 09:43:41,732 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:43:41,733 INFO exited: 71-cloudera-mgmt-SERVICEMONITOR (exit status 1; not expected)
2016-01-21 09:43:42,765 INFO spawned: '71-cloudera-mgmt-SERVICEMONITOR' with pid 4215
2016-01-21 09:43:42,765 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:43:48,144 INFO exited: 68-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:43:49,784 INFO exited: 71-cloudera-mgmt-SERVICEMONITOR (exit status 1; not expected)
2016-01-21 09:43:50,787 INFO spawned: '68-cloudera-mgmt-HOSTMONITOR' with pid 4433
2016-01-21 09:43:50,788 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:43:51,822 INFO spawned: '71-cloudera-mgmt-SERVICEMONITOR' with pid 4480
2016-01-21 09:43:51,823 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:43:53,851 INFO success: 67-cloudera-mgmt-ALERTPUBLISHER entered RUNNING state, process has stayed up for > than 20 seconds (startsecs)
2016-01-21 09:43:54,852 INFO success: 70-cloudera-mgmt-ACTIVITYMONITOR entered RUNNING state, process has stayed up for > than 20 seconds (startsecs)
2016-01-21 09:43:54,852 INFO success: 69-cloudera-mgmt-EVENTSERVER entered RUNNING state, process has stayed up for > than 20 seconds (startsecs)
2016-01-21 09:43:56,944 INFO exited: 68-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:43:58,051 INFO exited: 71-cloudera-mgmt-SERVICEMONITOR (exit status 1; not expected)
2016-01-21 09:44:00,056 INFO spawned: '68-cloudera-mgmt-HOSTMONITOR' with pid 4721
2016-01-21 09:44:00,056 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:44:01,091 INFO spawned: '71-cloudera-mgmt-SERVICEMONITOR' with pid 4768
2016-01-21 09:44:01,091 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:44:06,274 INFO exited: 68-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:44:07,276 INFO gave up: 68-cloudera-mgmt-HOSTMONITOR entered FATAL state, too many start retries too quickly
2016-01-21 09:44:07,327 INFO exited: 71-cloudera-mgmt-SERVICEMONITOR (exit status 1; not expected)
2016-01-21 09:44:07,327 INFO gave up: 71-cloudera-mgmt-SERVICEMONITOR entered FATAL state, too many start retries too quickly
2016-01-21 09:48:12,795 INFO spawned: '72-cloudera-mgmt-HOSTMONITOR' with pid 6881
2016-01-21 09:48:12,795 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:48:18,919 INFO exited: 72-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:48:19,923 INFO spawned: '72-cloudera-mgmt-HOSTMONITOR' with pid 6939
2016-01-21 09:48:19,923 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:48:27,097 INFO exited: 72-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:48:29,103 INFO spawned: '72-cloudera-mgmt-HOSTMONITOR' with pid 7085
2016-01-21 09:48:29,103 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:48:35,700 INFO exited: 72-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:48:38,707 INFO spawned: '72-cloudera-mgmt-HOSTMONITOR' with pid 7171
2016-01-21 09:48:38,707 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:48:45,649 INFO exited: 72-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:48:46,651 INFO gave up: 72-cloudera-mgmt-HOSTMONITOR entered FATAL state, too many start retries too quickly
2016-01-21 09:49:32,303 INFO spawned: '73-cloudera-mgmt-HOSTMONITOR' with pid 7709
2016-01-21 09:49:32,303 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:49:38,499 INFO exited: 73-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:49:39,502 INFO spawned: '73-cloudera-mgmt-HOSTMONITOR' with pid 7796
2016-01-21 09:49:39,503 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:49:46,168 INFO exited: 73-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:49:48,616 INFO spawned: '73-cloudera-mgmt-HOSTMONITOR' with pid 7943
2016-01-21 09:49:48,617 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:49:56,207 INFO exited: 73-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:49:59,213 INFO spawned: '73-cloudera-mgmt-HOSTMONITOR' with pid 7999
2016-01-21 09:49:59,214 INFO Increased RLIMIT_NOFILE limit to 1024
2016-01-21 09:50:06,669 INFO exited: 73-cloudera-mgmt-HOSTMONITOR (exit status 1; not expected)
2016-01-21 09:50:07,670 INFO gave up: 73-cloudera-mgmt-HOSTMONITOR entered FATAL state, too many start retries too quickly
Cloudera Employee
Posts: 289
Registered: ‎07-08-2013

Re: Host Monitor Not Starting

If this is on Ubuntu, there's a know bug on that OS, can you set the ulimit to 32k (~32768) and restart HMON

 

Navigate to:

CM> MGMT> Host Monitor Default Group / Performance> Maximum Process File Descriptors
CM> MGMT> Service Monitor Default Group / Performance> Maximum Process File Descriptors

New Contributor
Posts: 3
Registered: ‎01-21-2016

Re: Host Monitor Not Starting

THanks but I Re-installed CDH and it fixed the issue. 

Announcements