Reply
Contributor
Posts: 42
Registered: ‎05-30-2015
Accepted Solution

Cannot start cluster and cloudera management service

Dear all,

 

I was changing the database of my newly setup cloudera 5.11 on Ubuntu 14.04 to Mysql from embedded postgre database and out of sudden my cluster and cloudera management service all couldn't start, I have tried to change the database back to embedded, it still the same, the namenode fails to start. Anyone had the same issue before?

 

The role log as follow:

 

HTTP ERROR 500

Problem accessing /cmf/role/45/logs. Reason:

    INTERNAL_SERVER_ERROR

 

Caused by:

java.lang.NullPointerException
	at org.springframework.web.servlet.view.RedirectView.appendQueryProperties(RedirectView.java:252)
	at org.springframework.web.servlet.view.RedirectView.renderMergedOutputModel(RedirectView.java:225)
	at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
	at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
	at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

 

Cloudera Employee
Posts: 6
Registered: ‎01-30-2017

Re: Cannot start cluster and cloudera management service

 

This "Problem accessing /cmf/role/45/logs" denotes a specific URL for viewing a log file.

Could you make sure to point the browser to /cmf ? Any errors for that page? Can you try and access the Recent Commands view and from there check if you are able to see relevant logs for that failing start command?

Posts: 352
Topics: 11
Kudos: 55
Solutions: 30
Registered: ‎09-02-2016

Re: Cannot start cluster and cloudera management service

@yongie

 

1. Can you please list the steps that you have followed to migrate from Postgres to MySQL

2. Please list the impacted services as part of this migration

3. I hope you are doing the configuration setup via Cloudera Manager, pls confim

 

Contributor
Posts: 42
Registered: ‎05-30-2015

Re: Cannot start cluster and cloudera management service

[ Edited ]

@saranvisa

 

I followed the guide provided on Cloudera documentation

 

1. install mysql

2. set the innodb and other settings as required.

3. create the database for all the services with user name and password

4. change the database setting in cloudera management

5. impacted services are: Hive, Hue, Oozie, rman, amon, nav, navms

6. I am doing it via Cloudera Manager.

Posts: 352
Topics: 11
Kudos: 55
Solutions: 30
Registered: ‎09-02-2016

Re: Cannot start cluster and cloudera management service

@yongie

 

1. install mysql
Different CDH versions will support different MySQL version. Did you install compatiable mySQL version for the corresponding CDH? Pls confirm

 

4. change the database setting in cloudera management
Pls list the database setting change that you have made in CM for any one service. Ex: Hive

 

5. impacted services are: Hive, Hue, Oozie, rman, amon, nav, navms
What is the status of non-impacted services like Zookeeper, HDFS, etc? Fix Cloudera Management services & Non-impacted services first then focus on impacted services

 

Contributor
Posts: 42
Registered: ‎05-30-2015

Re: Cannot start cluster and cloudera management service

@pmilani

 

on /cmf it shows 

 

HTTP ERROR 404
Problem accessing /cmf. Reason:

NOT_FOUND

 

on /cmf/home

Unable to issue query: the Host Monitor is not running

 

Under recent Command, the log looks exactly the same as what I posted

 

 

Cloudera Employee
Posts: 6
Registered: ‎01-30-2017

Re: Cannot start cluster and cloudera management service

Sounds like the web server is up but pretty much failing severely.

Then could you check log files in /var/log/cloudera-scm-server/ to further troubleshoot the server start failure

Contributor
Posts: 42
Registered: ‎05-30-2015

Re: Cannot start cluster and cloudera management service

@saranvisa

 

1. I was using Mysql 5.5 initially, until it breaks, I thought it could be Mysql version, I upgraded to version 5.6, I am running CDH 5.11

 

2. Under Hive metastore database configuration, I select Mysql and changed the port from initially 7432 to 3306 on the same server.

 

3. Oh actually all services are affected, sorry, when I click on the spanar look icon, all services says:

(namenode1): No host heartbeat; CDH versions cannot be verified.

 

cybercdh.PNG

Contributor
Posts: 42
Registered: ‎05-30-2015

Re: Cannot start cluster and cloudera management service

[ Edited ]

@pmilani

 

I saw this when I try to start the Cloudera Management service.

 

2017-05-19 16:56:36,134 WARN 538954474@scm-web-313:com.cloudera.server.web.cmf.StatusProvider: (138 skipped) Failed to submit task for getting status from SERVICE_MONITORING
com.cloudera.cmon.MgmtServiceNotRunningException: SERVICE_MONITORING is not running

Highlighted
Cloudera Employee
Posts: 6
Registered: ‎01-30-2017

Re: Cannot start cluster and cloudera management service

that line is marked 'WARN', that would not cause all the malfunction you are experiencing. You should look into the rest of server logs.

To ensure the problem lies with the database, can you backup your DB and try creating a new empty DB and start the CM server?

Assuming configuration is correct, this should work and eventually drop you into the add cluster wizard when you point your browser to the server

Announcements