Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

cannot start Cloudera manager service

cannot start Cloudera manager service

Explorer

Hi,

I can't start cloudera manager service .Env: cdh 5.14.4 ubuntu 16.04.6 . 

6 REPLIES 6

Re: cannot start Cloudera manager service

Explorer

The error says 

Command aborted because of exception: Command timed-out after 150 seconds

Another error is 

HTTP ERROR 500

Problem accessing /cmf/role/27/logs. Reason: INTERNAL_SERVER_ERROR .

When I use spark2-submit to run a job on terminal, it says:

19/10/08 21:21:39 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/172.168.50.171:8032

19/10/08 21:21:40 INFO ipc.Client: Retrying connect to server: hadoop-master/172.168.50.171:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

 

Re: cannot start Cloudera manager service

Community Manager

Hi @susan_bj ,

 

What is your Cloudera Manager version? Could you please share a screenshot of the error to us? Where do you see the HTTP ERROR 500? Was this working before? Is there anything changed on the cluster?

 

Also if you go to the Instances page under Cloudera Management Service, do you see all of the roles not started? What if you try to start them one by one?

 

I would suggest you attaching Cloudera manager agent logs from the failing hosts so we can try to find out more data to work with.

 

Thanks,

Li

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Re: cannot start Cloudera manager service

Explorer

What is your Cloudera Manager version? CDH5.14.4

 Could you please share a screenshot of the error to us? 

Here is:

CDHerror.jpg

Where do you see the HTTP ERROR 500?

I click the full log file from the above picture.

 Was this working before?  Yes, it worked well before.

Is there anything changed on the cluster?  After I installed CDH ,it worked fine. Then I add some users on Linux, installed some libraries for Python,and copied some HDFS files from other cluster.

Highlighted

Re: cannot start Cloudera manager service

Explorer

Continue with the reply.

Also if you go to the Instances page under Cloudera Management Service, do you see all of the roles not started? What if you try to start them one by one?

None of the Instances started, when I tried to start them one by one, errors occurs, and they are the same as I tried to start Cloudera Manager Service.

 

I would suggest you attaching Cloudera manager agent logs from the failing hosts so we can try to find out more data to work with.

One log says:

[09/Oct/2019 09:13:44 +0000] 4819 MonitorDaemon-Reporter throttling_logger ERROR    (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-e3dc96ab2a3f384d8dace815d2dddeaf

Traceback (most recent call last):

  File "/opt/cm-5.14.4/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.4-py2.7.egg/cmf/monitor/firehose.py", line 120, in _send

    self._port)

  File "/opt/cm-5.14.4/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__

    self.conn.connect()

  File "/usr/lib/python2.7/httplib.py", line 846, in connect

    self.timeout, self.source_address)

  File "/usr/lib/python2.7/socket.py", line 575, in create_connection

    raise err

error: [Errno 111] Connection refused

Re: cannot start Cloudera manager service

Explorer

Another error is that the master node can't recognize by CDH, as the picture shows( each memory in the cluster is 8G).

CDHerr2.jpg

Re: cannot start Cloudera manager service

Explorer

The following is the error when I try start host monitor role:

2019-10-09 14:43:10,781 ERROR 1251477446@scm-web-154:org.mortbay.log: /cmf/role/29/logs

java.lang.NullPointerException

at org.springframework.web.servlet.view.RedirectView.appendQueryProperties(RedirectView.java:252)

at org.springframework.web.servlet.view.RedirectView.renderMergedOutputModel(RedirectView.java:225)

at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)

at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)

at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)

at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)

at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)

at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)

at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)

at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Don't have an account?
Coming from Hortonworks? Activate your account here