Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot start cluster and cloudera management service

Solved Go to solution

Cannot start cluster and cloudera management service

Rising Star

Dear all,

 

I was changing the database of my newly setup cloudera 5.11 on Ubuntu 14.04 to Mysql from embedded postgre database and out of sudden my cluster and cloudera management service all couldn't start, I have tried to change the database back to embedded, it still the same, the namenode fails to start. Anyone had the same issue before?

 

The role log as follow:

 

HTTP ERROR 500

Problem accessing /cmf/role/45/logs. Reason:

    INTERNAL_SERVER_ERROR

 

Caused by:

java.lang.NullPointerException
	at org.springframework.web.servlet.view.RedirectView.appendQueryProperties(RedirectView.java:252)
	at org.springframework.web.servlet.view.RedirectView.renderMergedOutputModel(RedirectView.java:225)
	at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
	at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
	at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Cannot start cluster and cloudera management service

Rising Star

Managed to solve it by copying the config.ini file from other servers form the same cluster and install the cloudera-scm-agent using apt-get install cloudera-scm-agent

11 REPLIES 11

Re: Cannot start cluster and cloudera management service

Cloudera Employee

 

This "Problem accessing /cmf/role/45/logs" denotes a specific URL for viewing a log file.

Could you make sure to point the browser to /cmf ? Any errors for that page? Can you try and access the Recent Commands view and from there check if you are able to see relevant logs for that failing start command?

Re: Cannot start cluster and cloudera management service

Rising Star

@pmilani

 

on /cmf it shows 

 

HTTP ERROR 404
Problem accessing /cmf. Reason:

NOT_FOUND

 

on /cmf/home

Unable to issue query: the Host Monitor is not running

 

Under recent Command, the log looks exactly the same as what I posted

 

 

Re: Cannot start cluster and cloudera management service

Cloudera Employee

Sounds like the web server is up but pretty much failing severely.

Then could you check log files in /var/log/cloudera-scm-server/ to further troubleshoot the server start failure

Re: Cannot start cluster and cloudera management service

Rising Star

@pmilani

 

I saw this when I try to start the Cloudera Management service.

 

2017-05-19 16:56:36,134 WARN 538954474@scm-web-313:com.cloudera.server.web.cmf.StatusProvider: (138 skipped) Failed to submit task for getting status from SERVICE_MONITORING
com.cloudera.cmon.MgmtServiceNotRunningException: SERVICE_MONITORING is not running

Re: Cannot start cluster and cloudera management service

Cloudera Employee

that line is marked 'WARN', that would not cause all the malfunction you are experiencing. You should look into the rest of server logs.

To ensure the problem lies with the database, can you backup your DB and try creating a new empty DB and start the CM server?

Assuming configuration is correct, this should work and eventually drop you into the add cluster wizard when you point your browser to the server

Highlighted

Re: Cannot start cluster and cloudera management service

Rising Star

@pmilani

 

Seems like the cloudera-scm-agent is not recognized as service, it was previously working.

 

service cloudera-scm-agent start
cloudera-scm-agent: unrecognized service

Re: Cannot start cluster and cloudera management service

Rising Star

Managed to solve it by copying the config.ini file from other servers form the same cluster and install the cloudera-scm-agent using apt-get install cloudera-scm-agent

Re: Cannot start cluster and cloudera management service

Champion

@yongie

 

1. Can you please list the steps that you have followed to migrate from Postgres to MySQL

2. Please list the impacted services as part of this migration

3. I hope you are doing the configuration setup via Cloudera Manager, pls confim

 

Re: Cannot start cluster and cloudera management service

Rising Star

@saranvisa

 

I followed the guide provided on Cloudera documentation

 

1. install mysql

2. set the innodb and other settings as required.

3. create the database for all the services with user name and password

4. change the database setting in cloudera management

5. impacted services are: Hive, Hue, Oozie, rman, amon, nav, navms

6. I am doing it via Cloudera Manager.