Support Questions

ask_bill_brooks · ‎10-08-2019

Hi,

I can't start cloudera manager service .Env: cdh 5.14.4 ubuntu 16.04.6 .

susan_bj · ‎10-08-2019

The error says

Command aborted because of exception: Command timed-out after 150 seconds

Another error is

HTTP ERROR 500

Problem accessing /cmf/role/27/logs. Reason: INTERNAL_SERVER_ERROR .

When I use spark2-submit to run a job on terminal, it says:

19/10/08 21:21:39 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/172.168.50.171:8032

19/10/08 21:21:40 INFO ipc.Client: Retrying connect to server: hadoop-master/172.168.50.171:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

lwang · ‎10-08-2019

Hi @susan_bj ,

What is your Cloudera Manager version? Could you please share a screenshot of the error to us? Where do you see the HTTP ERROR 500? Was this working before? Is there anything changed on the cluster?

Also if you go to the Instances page under Cloudera Management Service, do you see all of the roles not started? What if you try to start them one by one?

I would suggest you attaching Cloudera manager agent logs from the failing hosts so we can try to find out more data to work with.

Thanks,

Li

Li Wang, Technical Solution Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

susan_bj · ‎10-08-2019

What is your Cloudera Manager version? CDH5.14.4

Could you please share a screenshot of the error to us?

Here is:

Where do you see the HTTP ERROR 500?

I click the full log file from the above picture.

Was this working before? Yes, it worked well before.

Is there anything changed on the cluster? After I installed CDH ,it worked fine. Then I add some users on Linux, installed some libraries for Python,and copied some HDFS files from other cluster.

susan_bj · ‎10-08-2019

Continue with the reply.

Also if you go to the Instances page under Cloudera Management Service, do you see all of the roles not started? What if you try to start them one by one?

None of the Instances started, when I tried to start them one by one, errors occurs, and they are the same as I tried to start Cloudera Manager Service.

I would suggest you attaching Cloudera manager agent logs from the failing hosts so we can try to find out more data to work with.

One log says:

[09/Oct/2019 09:13:44 +0000] 4819 MonitorDaemon-Reporter throttling_logger ERROR (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-e3dc96ab2a3f384d8dace815d2dddeaf

Traceback (most recent call last):

File "/opt/cm-5.14.4/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.4-py2.7.egg/cmf/monitor/firehose.py", line 120, in _send

self._port)

File "/opt/cm-5.14.4/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__

self.conn.connect()

File "/usr/lib/python2.7/httplib.py", line 846, in connect

self.timeout, self.source_address)

File "/usr/lib/python2.7/socket.py", line 575, in create_connection

raise err

error: [Errno 111] Connection refused

susan_bj · ‎10-08-2019

Another error is that the master node can't recognize by CDH, as the picture shows( each memory in the cluster is 8G).

susan_bj · ‎10-09-2019

The following is the error when I try start host monitor role:

2019-10-09 14:43:10,781 ERROR 1251477446@scm-web-154:org.mortbay.log: /cmf/role/29/logs

java.lang.NullPointerException

at org.springframework.web.servlet.view.RedirectView.appendQueryProperties(RedirectView.java:252)

at org.springframework.web.servlet.view.RedirectView.renderMergedOutputModel(RedirectView.java:225)

at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)

at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)

at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)

at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)

at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)