Member since
10-08-2019
10
Posts
0
Kudos Received
0
Solutions
10-12-2019
06:43 AM
After I start the server, I saw some warnings in cloudera-scm-server.out file, it says"
WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. " I wonder if I can omit this warning or not.
I also found some errors from cloudera-scm-server.log ,it says"
CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: (29 skipped) Failed to send metrics.
org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connection refused)"
mysql version: mysql Ver 14.14 Distrib 5.7.27, CDH version 5.14.4, OS is Ubuntu 16.04.6
... View more
Labels:
10-11-2019
03:05 AM
Let me give more information:the CDH version is 5.14.4, and the OS system is Ubuntu 16.04.6. The cloudera-scm-agent.log says: MonitorDaemon-Reporter throttling_logger ERROR (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-e3dc96ab2a3f384d8dace815d2dddeaf Traceback (most recent call last): File "/opt/cm-5.14.4/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.4-py2.7.egg/cmf/monitor/firehose.py", line 120, in _send self._port) File "/opt/cm-5.14.4/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__ self.conn.connect() File "/usr/lib/python2.7/httplib.py", line 846, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 575, in create_connection raise err
... View more
10-11-2019
02:27 AM
I expand the server's memory to 16G, and each agent still has 8G. I did the following steps to start up the cluster: (1)stop all the agents by using the command: /opt/cm-5.14.4/etc/init.d/cloudera-scm-agent stop (2) stop the server :/opt/cm-5.14.4/etc/init.d/cloudera-scm-server stop (3)start server: /opt/cm-5.14.4/etc/init.d/cloudera-scm-server start (4)start agents:/opt/cm-5.14.4/etc/init.d/cloudera-scm-agent start But the problem remains the same. The cloudera-scm-server.log says: INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: Failed to send metrics. java.lang.reflect.UndeclaredThrowableException at com.sun.proxy.$Proxy113.writeMetrics(Unknown Source) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connection refused) at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88) ... 11 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) at sun.net.www.http.HttpClient.New(HttpClient.java:339) at sun.net.www.http.HttpClient.New(HttpClient.java:357) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315) at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72) at org.apache.avro.ipc.Requestor.request(Requestor.java:147) at org.apache.avro.ipc.Requestor.request(Requestor.java:101) at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72) ... 11 more Please help me to solve this puzzling problem.
... View more
- Tags:
- cloudera manager
10-09-2019
06:59 PM
I set up 4 machines (Ubuntu 16.04, CDH 5.14.4 ) as a cluster, but now, after I start them, the master node can't be recognized by Cloudera. Each machine has 8G memory.
I also have some other problems, like cloudera manager service can't start. The more detail is posted in this post:https://community.cloudera.com/t5/Support-Questions/cannot-start-Cloudera-manager-service/m-p/279713#M208409
... View more
Labels:
10-09-2019
12:01 AM
The following is the error when I try start host monitor role: 2019-10-09 14:43:10,781 ERROR 1251477446@scm-web-154:org.mortbay.log: /cmf/role/29/logs java.lang.NullPointerException at org.springframework.web.servlet.view.RedirectView.appendQueryProperties(RedirectView.java:252) at org.springframework.web.servlet.view.RedirectView.renderMergedOutputModel(RedirectView.java:225) at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250) at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574) at javax.servlet.http.HttpServlet.service(HttpServlet.java:575) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78) at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
... View more
10-08-2019
08:04 PM
Another error is that the master node can't recognize by CDH, as the picture shows( each memory in the cluster is 8G).
... View more
10-08-2019
07:59 PM
Continue with the reply. Also if you go to the Instances page under Cloudera Management Service, do you see all of the roles not started? What if you try to start them one by one? None of the Instances started, when I tried to start them one by one, errors occurs, and they are the same as I tried to start Cloudera Manager Service. I would suggest you attaching Cloudera manager agent logs from the failing hosts so we can try to find out more data to work with. One log says: [09/Oct/2019 09:13:44 +0000] 4819 MonitorDaemon-Reporter throttling_logger ERROR (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-e3dc96ab2a3f384d8dace815d2dddeaf Traceback (most recent call last): File "/opt/cm-5.14.4/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.4-py2.7.egg/cmf/monitor/firehose.py", line 120, in _send self._port) File "/opt/cm-5.14.4/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__ self.conn.connect() File "/usr/lib/python2.7/httplib.py", line 846, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 575, in create_connection raise err error: [Errno 111] Connection refused
... View more
10-08-2019
07:57 PM
What is your Cloudera Manager version? CDH5.14.4 Could you please share a screenshot of the error to us? Here is: Where do you see the HTTP ERROR 500? I click the full log file from the above picture. Was this working before? Yes, it worked well before. Is there anything changed on the cluster? After I installed CDH ,it worked fine. Then I add some users on Linux, installed some libraries for Python,and copied some HDFS files from other cluster.
... View more
10-08-2019
06:32 AM
The error says Command aborted because of exception: Command timed-out after 150 seconds Another error is HTTP ERROR 500 Problem accessing /cmf/role/27/logs. Reason: INTERNAL_SERVER_ERROR . When I use spark2-submit to run a job on terminal, it says: 19/10/08 21:21:39 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/172.168.50.171:8032 19/10/08 21:21:40 INFO ipc.Client: Retrying connect to server: hadoop-master/172.168.50.171:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
... View more
10-08-2019
06:23 AM
Hi,
I can't start cloudera manager service .E nv: cdh 5.14.4 ubuntu 16.04.6 .
... View more
Labels: