Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Standby NN not coming up

Explorer

Hi Team, client production cluster 3 Weeks back suddenly down.That time they started the cluster services all services are came up but roll change happened and Standby name node is active and cluster is up and running.while starting the other name node(previous active name node) not coming up and strucked at "RPC wait" and in CM for that namenode status showing as 'BUSY'.if any one help me on this issue

1 ACCEPTED SOLUTION

Contributor

Can you please check zookeeper logs once and share if any errors in the zookeeper logs

View solution in original post

4 REPLIES 4

Explorer

Standby NN logfile:

 

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:322)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.UnsupportedOperationException: CollectionUsage threshold is not supported
at sun.management.MemoryPoolImpl.getCollectionUsageThreshold(MemoryPoolImpl.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

.

.

.

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.UnsupportedOperationException: Usage threshold is not supported
at sun.management.MemoryPoolImpl.getUsageThresholdCount(MemoryPoolImpl.java:189)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Explorer

FAILOVERCONTROLLER logfile of Standby NN(previous ACTIVE NN):

 

2021-10-13 04:35:26,628 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at passive-mns-vm1/10.4.185.90:8022: java.net.ConnectException: Connection refused Call From passive-mns-vm1/10.4.185.90 to passive-mns-vm1:8022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
2021-10-13 04:35:28,628 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: passive-mns-vm1/10.4.185.90:8022. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
2021-10-13 04:35:28,629 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at passive-mns-vm1/10.4.185.90:8022: java.net.ConnectException: Connection refused Call From passive-mns-vm1/10.4.185.90 to passive-mns-vm1:8022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

Explorer

Could you please help me on this issue??

Contributor

Can you please check zookeeper logs once and share if any errors in the zookeeper logs