question Re: Cannot start cluster and cloudera management service in Support Questions

Cannot start cluster and cloudera management service

yongie — Thu, 18 May 2017 08:16:32 GMT

Dear all,

I was changing the database of my newly setup cloudera 5.11 on Ubuntu 14.04 to Mysql from embedded postgre database and out of sudden my cluster and cloudera management service all couldn't start, I have tried to change the database back to embedded, it still the same, the namenode fails to start. Anyone had the same issue before?

The role log as follow:

HTTP ERROR 500

Problem accessing /cmf/role/45/logs. Reason:

    INTERNAL_SERVER_ERROR

Caused by:

java.lang.NullPointerException
	at org.springframework.web.servlet.view.RedirectView.appendQueryProperties(RedirectView.java:252)
	at org.springframework.web.servlet.view.RedirectView.renderMergedOutputModel(RedirectView.java:225)
	at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
	at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
	at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Re: Cannot start cluster and cloudera management service

pmilani — Thu, 18 May 2017 13:57:15 GMT

This "Problem accessing /cmf/role/45/logs" denotes a specific URL for viewing a log file.

Could you make sure to point the browser to /cmf ? Any errors for that page? Can you try and access the Recent Commands view and from there check if you are able to see relevant logs for that failing start command?

Re: Cannot start cluster and cloudera management service

saranvisa — Thu, 18 May 2017 14:04:06 GMT

@yongie

1. Can you please list the steps that you have followed to migrate from Postgres to MySQL

2. Please list the impacted services as part of this migration

3. I hope you are doing the configuration setup via Cloudera Manager, pls confim

Re: Cannot start cluster and cloudera management service

yongie — Fri, 19 May 2017 00:57:38 GMT

@saranvisa

I followed the guide provided on Cloudera documentation

1. install mysql

2. set the innodb and other settings as required.

3. create the database for all the services with user name and password

4. change the database setting in cloudera management

5. impacted services are: Hive, Hue, Oozie, rman, amon, nav, navms

6. I am doing it via Cloudera Manager.

Re: Cannot start cluster and cloudera management service

saranvisa — Fri, 19 May 2017 04:01:58 GMT

@yongie

1. install mysql
Different CDH versions will support different MySQL version. Did you install compatiable mySQL version for the corresponding CDH? Pls confirm

4. change the database setting in cloudera management
Pls list the database setting change that you have made in CM for any one service. Ex: Hive

5. impacted services are: Hive, Hue, Oozie, rman, amon, nav, navms
What is the status of non-impacted services like Zookeeper, HDFS, etc? Fix Cloudera Management services & Non-impacted services first then focus on impacted services

Re: Cannot start cluster and cloudera management service

yongie — Fri, 19 May 2017 08:21:17 GMT

@pmilani

on /cmf it shows

HTTP ERROR 404
Problem accessing /cmf. Reason:

NOT_FOUND

on /cmf/home

Unable to issue query: the Host Monitor is not running

Under recent Command, the log looks exactly the same as what I posted

Re: Cannot start cluster and cloudera management service

pmilani — Fri, 19 May 2017 08:27:55 GMT

Sounds like the web server is up but pretty much failing severely.

Then could you check log files in /var/log/cloudera-scm-server/ to further troubleshoot the server start failure

Re: Cannot start cluster and cloudera management service

yongie — Fri, 19 May 2017 08:36:51 GMT

@saranvisa

1. I was using Mysql 5.5 initially, until it breaks, I thought it could be Mysql version, I upgraded to version 5.6, I am running CDH 5.11

2. Under Hive metastore database configuration, I select Mysql and changed the port from initially 7432 to 3306 on the same server.

3. Oh actually all services are affected, sorry, when I click on the spanar look icon, all services says:

(namenode1): No host heartbeat; CDH versions cannot be verified.

Re: Cannot start cluster and cloudera management service

yongie — Fri, 19 May 2017 09:29:12 GMT

@pmilani

I saw this when I try to start the Cloudera Management service.

2017-05-19 16:56:36,134 WARN 538954474@scm-web-313:com.cloudera.server.web.cmf.StatusProvider: (138 skipped) Failed to submit task for getting status from SERVICE_MONITORING
com.cloudera.cmon.MgmtServiceNotRunningException: SERVICE_MONITORING is not running

Re: Cannot start cluster and cloudera management service

pmilani — Fri, 19 May 2017 10:39:39 GMT

that line is marked 'WARN', that would not cause all the malfunction you are experiencing. You should look into the rest of server logs.

To ensure the problem lies with the database, can you backup your DB and try creating a new empty DB and start the CM server?

Assuming configuration is correct, this should work and eventually drop you into the add cluster wizard when you point your browser to the server

Re: Cannot start cluster and cloudera management service

yongie — Mon, 22 May 2017 03:23:36 GMT

@pmilani

Seems like the cloudera-scm-agent is not recognized as service, it was previously working.

service cloudera-scm-agent start
cloudera-scm-agent: unrecognized service

Re: Cannot start cluster and cloudera management service

yongie — Wed, 24 May 2017 01:35:50 GMT

Managed to solve it by copying the config.ini file from other servers form the same cluster and install the cloudera-scm-agent using apt-get install cloudera-scm-agent