Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

CM5 Service Monitor Failed & Could Not Connect to Host Monitor Errors

New Contributor

Hi,

 

I installed CDH5 using installation path A and everthing is working with the exception of the monitoring.  

 

All services have "Unknown Health", charts have the error of "Unable to issue query: could not connect to the Host Monitor", the top of the home page has "Request to the Service Monitor failed. This may cause slow page responses".  The last error when clicking into it returns a 404 error.

 

Looking at the logs I'm seeing the following warnings:

1 - WARN [1752771961@scm-web-7:cmf.StatusProvider@503] Failed to get status.

2 - WARN [1202997703@scm-web-1:tsquery.TimeSeriesQueryService@503] Could not find a HOST_MONITORING nozzle from SCM.

3 - WARN [44428231@scm-web-34:cmf.EventsController@503] Exception querying events com.cloudera.cmon.MgmtServiceLocatorException: There is no configured event store

 

I'd appreciate any insight.

 

Thanks - J

12 REPLIES 12

Master Guru
Can you post your entire Service and Host monitor logs here, or via pastebin links?

New Contributor

this is entire Service and Host monitor logs

ServerLog

I don't have a QQ account. Please use a service like pastebin that doesn't
need us to sign in.

Regards,
Gautam Gopalakrishnan

New Contributor

Hi, i have the same problem,did you have resolve it ? very thanks~!


 

Expert Contributor

i got the same error,

 

Restart your Cloudera management services,it will get working.

 

 

Explorer

I am getting the same issue. However, I am not getting the option of cloudera management services to restart on UI< attached is the screenshot. Do we need to install and configure anything else apart from cloudera-scm-server and cloudera-scm-agent?

 

swapko_0-1617785250343.png

 

Community Manager

@swapko as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

New Contributor
Does your CM work well? I met the same truble: 2014-11-13 21:39:24,485 WARN 460709139@scm-web-34:com.cloudera.server.cmf.tsquery.TimeSeriesQueryService: (107 skipped) Could not find a HOST_MONITORING nozzle from SCM. com.cloudera.cmon.MgmtServiceLocatorException: Could not find a HOST_MONITORING nozzle from SCM. at com.cloudera.cmon.MgmtServiceLocator.getNozzleIPC(MgmtServiceLocator.java:141) at com.cloudera.server.cmf.tsquery.NozzleRequest.(NozzleRequest.java:50) at com.cloudera.server.cmf.tsquery.TimeSeriesMultiRequest.(TimeSeriesMultiRequest.java:48) at com.cloudera.server.cmf.tsquery.TimeSeriesQueryService.queryTimeSeries(TimeSeriesQueryService.java:306) at com.cloudera.server.web.cmf.charts.TimeSeriesQueryController.queryTimeSeriesHelper(TimeSeriesQueryController.java:303) at com.cloudera.server.web.cmf.charts.TimeSeriesQueryController.queryTimeSeries(TimeSeriesQueryController.java:269) at sun.reflect.GeneratedMethodAccessor808.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:585) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) …… ----------------------------------------------------------------------------------------------------------------------------------------------- I really don't know why? If anyone can tell me why ,I would be grateful.

New Contributor

Explorer

i user cm version 5.4.5, after i reboot the cluster, i also has the same problem. The admin console output "Request to the Service Monitor failed. This may cause slow page responses. View the status of the Service Monitor.", who can solve it

New Contributor

This may sound like a silly solution and I'm sure will not fix everyones problem but I was able to get both my host & service monitors back up and running with a simple restart of the monitors.

This has worked for me in 2 diffent CDH managed environments.

 

The question now is what caused them to fail in the first place?

 

New Contributor

I've check the log file under /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-hadoop1.log.out and found the error below

 

2016-04-26 14:43:27,069 INFO com.cloudera.cmon.tstore.leveldb.LDBUtils: Creating directory /var/lib/cloudera-service-monitor/ts/ts_entity_metadata
2016-04-26 14:43:27,072 ERROR com.cloudera.cmon.firehose.Main: Error creating LevelDB timeseries store in directory /var/lib/cloudera-service-monitor/ts
java.io.IOException: Unable to create new directory at /var/lib/cloudera-service-monitor/ts/ts_entity_metadata
at com.cloudera.cmon.tstore.leveldb.LDBUtils.openVersionedDB(LDBUtils.java:239)
at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesMetadataStore.openMetadataDB(LDBTimeSeriesMetadataStore.java:149)
at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesMetadataStore.<init>(LDBTimeSeriesMetadataStore.java:139)
at com.cloudera.cmon.firehose.Main.main(Main.java:450)

 

Creating /var/lib/cloudera-service-monitor and /var/lib/cloudera-host-monitor directories manually solved my issue.