Reply
Contributor
Posts: 28
Registered: ‎03-04-2014

Cloudera Manager often goes down

All,

 

Cloudera-scm-server often goes down . following is the error message that i see  while bringing up the Cloudera Manager.

 

2014-05-20 11:03:57,933  WARN [WebServerImpl:components.SearchRepositoryManager@181] Failed to initialize search dir, deleting it: /var/lib/cloudera-scm-server/search/lucene.en..1400608940959
org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.NIOFSDirectory@/var/lib/cloudera-scm-server/search/lucene.en..1400608940959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@437024b0: files: []
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:694)
        at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
        at org.apache.lucene.index.IndexReader.open(IndexReader.java:428)
        at org.apache.lucene.index.IndexReader.open(IndexReader.java:288)
        at org.apache.lucene.search.IndexSearcher.<init>(IndexSearcher.java:82)
        at com.cloudera.server.web.cmf.search.LuceneSearchRepository.<init>(LuceneSearchRepository.java:106)
        at com.cloudera.server.web.cmf.search.components.SearchRepositoryManager.initializeState(SearchRepositoryManager.java:167)
        at com.cloudera.server.web.cmf.search.components.SearchRepositoryManager.<init>(SearchRepositoryManager.java:143)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:108)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:280)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
        at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:467)
        at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:483)
        at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:358)
        at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:325)
        at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:127)
        at javax.servlet.GenericServlet.init(GenericServlet.java:241)
        at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
        at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:736)
        at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
        at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
        at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at com.cloudera.server.cmf.WebServerImpl.run(WebServerImpl.java:273)
2014-05-20 11:03:59,255 ERROR [WebServerImpl:cmf.TsqueryAutoCompleter@391] Error getting predicates
org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused

 

2014-05-20 11:04:30,148  WARN [2143114129@scm-web-1:cmf.StatusProvider@503] Failed to get status.
java.util.concurrent.ExecutionException: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused

 

I have the firewall turned off . Please help

Contributor
Posts: 28
Registered: ‎03-04-2014

Re: Cloudera Manager often goes down

I was able to resolve this issue, after fixing the configuration errors pertaining to Cloudera Management Service(mgmt). I had 2 memory related config issue on Java_Heap_Usage and nonjava_Heap_Usage which was misconfigured , that lead to mgmt service keep failing that inturn brings the cloudera manager down as scm unable to connect services SERVICE_MONITORING & HOST_MONITORING.

Highlighted
New Contributor
Posts: 2
Registered: ‎09-28-2017

Re: Cloudera Manager often goes down

Hi I got the same problem, Unable to resolve it. Can you please be more specific what you have done to resovle this problem. Where can i change Java_Heap_Size from terminal. 

Thanks in Advance.

Announcements