Reply
New Contributor
Posts: 2
Registered: ‎02-22-2015

Cloudera Manager won't start

Hi,

 

I'm running CDH5 on VM. WHen I try to start cloudera manager page on the VM throuh mozilla firefox home page, it won't start and hanging at the loading page. The message on the page is  below

 

Loading

Please wait while cloudera manager starts services

This may take a couple of minutes. You will be

forwarded to the login page when your cluster is ready.

 

Its like this for a long time. Restarted the guest from the menu but still its the same issue. Below is the log from the command

sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

 

Log is as below:

 

[cloudera@localhost ~]$ sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
2015-02-22 09:22:55,806 INFO [WebServerImpl:cmf.WebServerImpl@292] Started Jetty server.
2015-02-22 09:22:56,086 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@411] Num docs:3118
2015-02-22 09:22:56,090 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@352] Constructing repo:2015-02-22T15:22:56.090Z
2015-02-22 09:22:59,168 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@354] Finished constructing repo:2015-02-22T15:22:59.168Z
2015-02-22 09:23:03,786 INFO [JvmPauseMonitor:debug.JvmPauseMonitor@236] Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1119ms: GC pool 'PS Scavenge' had collection(s): count=1 time=1565ms
2015-02-22 09:23:03,787 INFO [JvmPauseMonitor:debug.JvmPauseMonitor@236] Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1120ms: GC pool 'PS Scavenge' had collection(s): count=1 time=1565ms
2015-02-22 09:32:14,916 INFO [StaleEntityEviction:model.HeartbeatStore@107] Reaped 0 process heartbeats and 0 host heartbeats.
2015-02-22 09:32:15,294 INFO [StaleEntityEviction:cmf.StaleEntityEvictionThread@238] Found no commands older than 2013-02-22T15:32:15.182Z to reap.
2015-02-22 09:32:15,297 INFO [StaleEntityEviction:node.NodeScannerService@335] Reaped 0 requests.
2015-02-22 09:32:15,298 INFO [StaleEntityEviction:node.NodeConfiguratorService@358] Reaped 0 requests.
2015-02-22 09:41:30,158 INFO [JvmPauseMonitor:debug.JvmPauseMonitor@236] Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1830ms: GC pool 'PS MarkSweep' had collection(s): count=1 time=2082ms, GC pool 'PS Scavenge' had collection(s): count=1 time=20ms
2015-02-22 09:41:30,158 INFO [JvmPauseMonitor:debug.JvmPauseMonitor@236] Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1835ms: GC pool 'PS MarkSweep' had collection(s): count=1 time=2082ms, GC pool 'PS Scavenge' had collection(s): count=1 time=20ms
2015-02-22 09:42:15,301 INFO [StaleEntityEviction:model.HeartbeatStore@107] Reaped 0 process heartbeats and 0 host heartbeats.
2015-02-22 09:42:15,334 INFO [StaleEntityEviction:cmf.StaleEntityEvictionThread@238] Found no commands older than 2013-02-22T15:42:15.326Z to reap.
2015-02-22 09:42:15,340 INFO [StaleEntityEviction:node.NodeScannerService@335] Reaped 0 requests.
2015-02-22 09:42:15,340 INFO [StaleEntityEviction:node.NodeConfiguratorService@358] Reaped 0 requests.
2015-02-22 09:52:15,344 INFO [StaleEntityEviction:model.HeartbeatStore@107] Reaped 0 process heartbeats and 0 host heartbeats.
2015-02-22 09:52:15,364 INFO [StaleEntityEviction:cmf.StaleEntityEvictionThread@238] Found no commands older than 2013-02-22T15:52:15.362Z to reap.
2015-02-22 09:52:15,366 INFO [StaleEntityEviction:node.NodeScannerService@335] Reaped 0 requests.
2015-02-22 09:52:15,366 INFO [StaleEntityEviction:node.NodeConfiguratorService@358] Reaped 0 requests.
2015-02-22 09:52:50,635 INFO [CMMetricsForwarder-0:components.ClouderaManagerMetricsForwarder@172] (29 skipped) Failed to send metrics.

 

java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy98.sendAgentMessages(Unknown Source)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:187)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:174)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:102)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
... 11 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)

 

 

Pleas help.

 

Ram

New Contributor
Posts: 2
Registered: ‎02-22-2015

Re: Cloudera Manager won't start

Really appreciate any help on this issue.

Highlighted
Explorer
Posts: 7
Registered: ‎02-24-2015

Re: Cloudera Manager won't start

Hi

 

I am trying CDH automatic installation on AWs EC2 using cloudera manager bin. I have created one ubuntu Precise 12.04 LTS micro instance,

 

I followed the on screen instructions as instructions on per this tutorial.. " http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v4-7-1/Cloudera-Manager-I...

 

1) this is my vi /etc/hosts file

127.0.0.1 localhost
172.31.13.46 master

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

 

2) Downloaded and changed the permission for cloudera-manager-installer.bin also.

 

3) sudo ./cloudera-manager-installer.bin after this command cloudera manager installed,

 

4) But i couldnot access cloudera manager webconsole using " http://172.31.13.46:7180 " and i have opened my port 7180 while creating an instance, but still not able to acess through webconsole, 

 

5) my cloudera manager db and cloudera manager server both are running.

 

6) and the port 7180 is also not listning in my ubuntu server and i used the following comand, " sudo ufs allow 7180"  but no use,,

 

7) I checked $ sudo ufw status and the result is inactive

 

8) when I check $ sudo service cloudera-scm-agent status on 172.31.13.46 it comes as unrecognized service

 

 

I am struggling in this part, Could you please let me know where I went wrong in installing cloudera in a clustered environment..???

 

if yes, it will be helpful for me, please,

 

Thanks in advance,

 

Regards,

Bharath

Contributor
Posts: 52
Registered: ‎10-19-2015

Re: Cloudera Manager won't start

We had the same issue in produciton. The CM Server jvm keeps hanging for GC, the jvm pause happens every few sec to few min. In a nomal status, the JVM pause happens every 6 to 10 hours.

 

Is this a bug in CM server. Our version is 5.5.1