Unable to issue query: the Host Monitor is not running

[#1]

I have a single node pseudo-distributed installation of Cloudera Express 5.5.1. I'm seeing "Unable to issue query: the Host Monitor is not running". When I try to restart the Management Service, all roles fail to start. Each log contains:

HTTP ERROR 500

Problem accessing /cmf/role/17/logs. Reason:

    INTERNAL_SERVER_ERROR

Caused by:

java.lang.NullPointerException
	at org.springframework.web.servlet.view.RedirectView.appendQueryProperties(RedirectView.java:252)
	at org.springframework.web.servlet.view.RedirectView.renderMergedOutputModel(RedirectView.java:225)
	at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
	at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
	at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Powered by Jetty://

varying only in the numeric subdirectory (eg. '17' in the path at the top ("/cmf/role/17/logs"). There is no 'role' directory anywhere on the system.

Additionally, there are no recent entries in the /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-HOSTMONITOR-kdahadoop.kda.local.log.out file, the last being almost 2 weeks ago.

Thanks in advance for any help.

[#2]

Hello,

I think may happen if Cloudera doesn't have enough resources (memory, CPU). What is your machine ?

I think that in order to have Cloudera Manager you need about at least 9 GB on your PC.

Check this out: https://community.cloudera.com/t5/Cloudera-Manager-Installation/CDH-5-1-2-Unable-to-issue-query-the-...

Hopes this helps,

[#3]

HI Alina,

     I'm running on a Lenovo ideapad Z710, which has 8GB of main memory and approximately 1TB of disk space.

If memory is the issue, it's a shame that CDH won't run on a machine such as this in some form or another, at least for research and development purposes.

[#4]

Hello,

If it's for test purposes, and since you have that much of disk, I suppose you could add some swap. That should help (not for the performances, but for having access to the platform).

Requirements: http://www.cloudera.com/downloads/quickstart_vms/5-4.html

If you count the OS; the browser that you may have open, the VM if you are running in a VM... I think you may be at the limit of memory.

[#5]

@megrez80 You can find the system requirements on the quickstart download page but I will show them below to speed things along. With your Lenovo system, you should be able to run CDH just fine as long as the VM meets the prerequisites, including at least 4GB or RAM dedicated to it. The problem is that when you run Cloudera Manager as well, the RAM requirement doubles. Additional RAM is also required if you are running workloads larger than the provided examples.  I hope this helps.

Prerequisites

• These 64-bit VMs require a 64-bit host OS and a virtualization product that can support a 64-bit guest OS.
• To use a VMware VM, you must use a player compatible with WorkStation 8.x or higher:
  • Player 4.x or higher
  • Fusion 4.x or higher

  Older versions of WorkStation can be used to create a new VM using the same virtual disk (VMDK file), but some features in VMware Tools are not available.

• The amount of RAM required varies by the run-time option you choose:

  CDH and Cloudera Manager Version	RAM Required by VM
  CDH 5 (default)	4+ GiB*
  Cloudera Express	8+ GiB*
  Cloudera Enterprise (trial)	10+ GiB*

  *Minimum recommended memory. If you are running workloads larger than the examples provided, consider allocating additional memory.

[#6]

CY,

     I'm not running in a VM, but directly on the host OS. Is 8GB enough in this case?

Thanks again.

[#7 · Accepted solution]

Oh, sorry for the assumption. I may need to ask around on this one since I am not an expert myself. 

My initial thoughts are that even installing without the use of the VM you may run into issues with only 8GB of total system RAM. Depending on how much of the RAM your PC takes to run, for example Windows 10 64 bit requires a minimum of 2GB of RAM , there may not be enough left over for CDH and Cloudera Manager to utilize.   

A little more information about your pseudo-distributed installation should be helpful. Can you explain a little more about your setup including the operating system and how you installed Cloudera Express?

Some additional reading material that may be helpful: Selecting the Right Hardware for Your New Hadoop Cluster. It doesn't diretly coordinate to your current setup but may be usefull when thinking of overall Hadoop requirements. 

[#8]

I'm running Ubuntu 14.04. I installed everything using the cloudera-manager-installer.bin.

What else would you like to know?

[#9]

@megrez80, my tendency here is to take a step back and look at the root cause of your error.  I agree that you are probably pushing the limits of what our system requirements are for testing, and maybe something like Cloudera Live would be a better option for a true POC, but I'm not convinced your problem is a memory problem.

The error clearly mentions that it can't open the logs directory and you stated that there is no /cmf/role/17/logs directory on your system, so let's start there.  Why did the directory not get created during installation?  There should be an install log located in your /tmp filesystem which captured what happened when CM was installing.

Forgive me for not remembering the exact file name, but it's something along the lines of scm-install.log.  If you can find that file, it might contain some valuable information as to why the correct directories were not created. 

[#10]

Clint. I can't find anything looking like an installation log in the /tmp directory. The machine has been rebooted a number of times since the install.