CM5 Service Monitor Failed & Could Not Connect to Host Monitor Errors

[#1]

Hi,

I installed CDH5 using installation path A and everthing is working with the exception of the monitoring.  

All services have "Unknown Health", charts have the error of "Unable to issue query: could not connect to the Host Monitor", the top of the home page has "Request to the Service Monitor failed. This may cause slow page responses".  The last error when clicking into it returns a 404 error.

Looking at the logs I'm seeing the following warnings:

1 - WARN [1752771961@scm-web-7:cmf.StatusProvider@503] Failed to get status.

2 - WARN [1202997703@scm-web-1:tsquery.TimeSeriesQueryService@503] Could not find a HOST_MONITORING nozzle from SCM.

3 - WARN [44428231@scm-web-34:cmf.EventsController@503] Exception querying events com.cloudera.cmon.MgmtServiceLocatorException: There is no configured event store

I'd appreciate any insight.

Thanks - J

[#2]

i user cm version 5.4.5, after i reboot the cluster, i also has the same problem. The admin console output "Request to the Service Monitor failed. This may cause slow page responses. View the status of the Service Monitor.", who can solve it

[#3]

This may sound like a silly solution and I'm sure will not fix everyones problem but I was able to get both my host & service monitors back up and running with a simple restart of the monitors.

This has worked for me in 2 diffent CDH managed environments.

The question now is what caused them to fail in the first place?

[#4]

I've check the log file under /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-hadoop1.log.out and found the error below

2016-04-26 14:43:27,069 INFO com.cloudera.cmon.tstore.leveldb.LDBUtils: Creating directory /var/lib/cloudera-service-monitor/ts/ts_entity_metadata
2016-04-26 14:43:27,072 ERROR com.cloudera.cmon.firehose.Main: Error creating LevelDB timeseries store in directory /var/lib/cloudera-service-monitor/ts
java.io.IOException: Unable to create new directory at /var/lib/cloudera-service-monitor/ts/ts_entity_metadata
at com.cloudera.cmon.tstore.leveldb.LDBUtils.openVersionedDB(LDBUtils.java:239)
at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesMetadataStore.openMetadataDB(LDBTimeSeriesMetadataStore.java:149)
at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesMetadataStore.<init>(LDBTimeSeriesMetadataStore.java:139)
at com.cloudera.cmon.firehose.Main.main(Main.java:450)

Creating /var/lib/cloudera-service-monitor and /var/lib/cloudera-host-monitor directories manually solved my issue.

[#5]

I run into same issue after deploying cdp 7.1.7. Is there workaround for this issue ?