question Re: Unable to issue query: the Host Monitor is not running in Support Questions

Unable to issue query: the Host Monitor is not running

megrez80 — Fri, 16 Sep 2022 10:05:38 GMT

I have a single node pseudo-distributed installation of Cloudera Express 5.5.1. I'm seeing "Unable to issue query: the Host Monitor is not running". When I try to restart the Management Service, all roles fail to start. Each log contains:

HTTP ERROR 500

Problem accessing /cmf/role/17/logs. Reason:

    INTERNAL_SERVER_ERROR

Caused by:

java.lang.NullPointerException
	at org.springframework.web.servlet.view.RedirectView.appendQueryProperties(RedirectView.java:252)
	at org.springframework.web.servlet.view.RedirectView.renderMergedOutputModel(RedirectView.java:225)
	at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
	at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
	at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Powered by Jetty://

varying only in the numeric subdirectory (eg. '17' in the path at the top ("/cmf/role/17/logs"). There is no 'role' directory anywhere on the system.

Additionally, there are no recent entries in the /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-kdahadoop.kda.local.log.out file, the last being almost 2 weeks ago.

Thanks in advance for any help.

Re: Unable to issue query: the Host Monitor is not running

AlinaGHERMAN — Thu, 25 Feb 2016 05:44:23 GMT

Hello,

I think may happen if Cloudera doesn't have enough resources (memory, CPU). What is your machine ?

I think that in order to have Cloudera Manager you need about at least 9 GB on your PC.

Check this out: https://community.cloudera.com/t5/Cloudera-Manager-Installation/CDH-5-1-2-Unable-to-issue-query-the-Host-Monitor-is-not-running/td-p/18404

Hopes this helps,

Re: Unable to issue query: the Host Monitor is not running

megrez80 — Thu, 25 Feb 2016 13:05:53 GMT

HI Alina,

I'm running on a Lenovo ideapad Z710, which has 8GB of main memory and approximately 1TB of disk space.

If memory is the issue, it's a shame that CDH won't run on a machine such as this in some form or another, at least for research and development purposes.

Re: Unable to issue query: the Host Monitor is not running

AlinaGHERMAN — Thu, 25 Feb 2016 13:31:27 GMT

Hello,

If it's for test purposes, and since you have that much of disk, I suppose you could add some swap. That should help (not for the performances, but for having access to the platform).

Requirements: http://www.cloudera.com/downloads/quickstart_vms/5-4.html

If you count the OS; the browser that you may have open, the VM if you are running in a VM... I think you may be at the limit of memory.

Re: Unable to issue query: the Host Monitor is not running

cjervis — Thu, 25 Feb 2016 13:58:08 GMT

@megrez80 You can find the system requirements on the quickstart download page but I will show them below to speed things along. With your Lenovo system, you should be able to run CDH just fine as long as the VM meets the prerequisites, including at least 4GB or RAM dedicated to it. The problem is that when you run Cloudera Manager as well, the RAM requirement doubles. Additional RAM is also required if you are running workloads larger than the provided examples. I hope this helps.

Prerequisites

These 64-bit VMs require a 64-bit host OS and a virtualization product that can support a 64-bit guest OS.
To use a VMware VM, you must use a player compatible with WorkStation 8.x or higher:
- Player 4.x or higher
- Fusion 4.x or higher
Older versions of WorkStation can be used to create a new VM using the same virtual disk (VMDK file), but some features in VMware Tools are not available.
The amount of RAM required varies by the run-time option you choose:

CDH and Cloudera Manager Version RAM Required by VM

CDH 5 (default) 4+ GiB*

Cloudera Express 8+ GiB*

Cloudera Enterprise (trial) 10+ GiB*

*Minimum recommended memory. If you are running workloads larger than the examples provided, consider allocating additional memory.

CDH and Cloudera Manager Version	RAM Required by VM
CDH 5 (default)	4+ GiB*
Cloudera Express	8+ GiB*
Cloudera Enterprise (trial)	10+ GiB*

Re: Unable to issue query: the Host Monitor is not running

megrez80 — Thu, 25 Feb 2016 15:04:15 GMT

CY,

I'm not running in a VM, but directly on the host OS. Is 8GB enough in this case?

Thanks again.

Re: Unable to issue query: the Host Monitor is not running

cjervis — Thu, 25 Feb 2016 15:40:19 GMT

Oh, sorry for the assumption. I may need to ask around on this one since I am not an expert myself.

My initial thoughts are that even installing without the use of the VM you may run into issues with only 8GB of total system RAM. Depending on how much of the RAM your PC takes to run, for example Windows 10 64 bit requires a minimum of 2GB of RAM , there may not be enough left over for CDH and Cloudera Manager to utilize.

A little more information about your pseudo-distributed installation should be helpful. Can you explain a little more about your setup including the operating system and how you installed Cloudera Express?

Some additional reading material that may be helpful: Selecting the Right Hardware for Your New Hadoop Cluster. It doesn't diretly coordinate to your current setup but may be usefull when thinking of overall Hadoop requirements.

Re: Unable to issue query: the Host Monitor is not running

megrez80 — Thu, 25 Feb 2016 16:53:44 GMT

I'm running Ubuntu 14.04. I installed everything using the cloudera-manager-installer.bin.

What else would you like to know?

Re: Unable to issue query: the Host Monitor is not running

Clint — Thu, 25 Feb 2016 17:44:43 GMT

@megrez80, my tendency here is to take a step back and look at the root cause of your error. I agree that you are probably pushing the limits of what our system requirements are for testing, and maybe something like Cloudera Live would be a better option for a true POC, but I'm not convinced your problem is a memory problem.

The error clearly mentions that it can't open the logs directory and you stated that there is no /cmf/role/17/logs directory on your system, so let's start there. Why did the directory not get created during installation? There should be an install log located in your /tmp filesystem which captured what happened when CM was installing.

Forgive me for not remembering the exact file name, but it's something along the lines of scm-install.log. If you can find that file, it might contain some valuable information as to why the correct directories were not created.

Re: Unable to issue query: the Host Monitor is not running

bgooley — Thu, 25 Feb 2016 17:48:11 GMT

I took a brief look at your issue. If Host Monitor is not running, then you can check the stderr.log and stdout.log for clues if it is running out of heap (in which case the default behavior is to shutdown the service).

In Cloudera Manager, go to Clusters --> Cloudera Management Service --> (under Status Summary) Host Monitor --> Processes

Click on the "stdout" link to open the stdout for the process. See if there are OOMEs there.

Regardless of how much RAM you have on this host, if the Host Monitor's max heap is tuned too low, it can happen that it will throw an OOME exception.

Re: Unable to issue query: the Host Monitor is not running

bgooley — Thu, 25 Feb 2016 17:50:58 GMT

Oops... disregard my last update. It was off topic!

Re: Unable to issue query: the Host Monitor is not running

megrez80 — Thu, 25 Feb 2016 18:09:59 GMT

Clint. I can't find anything looking like an installation log in the /tmp directory. The machine has been rebooted a number of times since the install.

Re: Unable to issue query: the Host Monitor is not running

megrez80 — Wed, 02 Mar 2016 13:46:54 GMT

This issue cropped up again, this time though I'm running CDH 5.6 in a 2 node vSphere ESXi-based cluster. Each node has 20GB of memory. Also different this time is a Management Service Restart was successfull.

This came after finally getting everything installed and started yesterday afternoon (a 3 node config failed multiple times to start all the services), and then this morning all the monitoring services had stopped.

What could cause this?

Re: Unable to issue query: the Host Monitor is not running

Herman Chen — Thu, 03 Mar 2016 07:24:23 GMT

What action triggered the stacktrace? The stacktrace is from deep within Spring and suggests system level issue, e.g. out of memory. A few things to check:

- server log (/var/log/cloudera-scm-server/cloudera-scm-server.log)

- management daemon logs (/var/log/cloudera-scm-firehose/*.log

- check "Hosts"->"All Hosts" for memory pressure. The "Resources" tab of individual Host page may help as well