Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP 3.1.1 YARN Resource manager starting fail

HDP 3.1.1 YARN Resource manager starting fail

New Contributor

Distribution HDP 3.1.1
Virtual machine with Centos 7.1
My cluster configuration have 5 node with this major services

 

h01: HDFS NAMENODE ACTIVE, YARN ResourceManager, TIMELINE SERVICE V2.0 READER, YARN REGISTRY DNS, HBase Client, ZOOKEEPER SERVER, HIVESERVER2
h02: HDFS NAMENODE STANDBY, YARN TIMELINE SERVER V1.5, YARN Client, ZOOKEEPER SERVER
h03: Datanode, NodeManager, YARN Client, HBase Client, ZOOKEEPER SERVER
h04: Datanode, NodeManager, YARN Client, HBase Client, ZOOKEEPER SERVER
h05: Datanode, NodeManager, YARN Client, HBase Client, ZOOKEEPER SERVER

 

I need to enhance CPU in h01 virtual machine from 4 to 6.
I stopped all services in h01. Shutdown. Updated to 6 CPU and Restart h01 virtual machine.
After restart, by ambari web GUI, i saw that following processes did not start
YARN ResourceManager, NodeManager, TIMELINE SERVICE V2.0 READER, TIMELINE SERVICE V2.0 READER, HIVESERVER2

Using Ambari Recover Host feature, i tried to recover h01 , but with no success.

Please, can anyone help me?

 

less /var/log/hadoop-yarn/yarn/hadoop-yarn-resourcemanager-node-xxxxx-h01.log

************************************************************/
2020-04-03 17:33:17,589 INFO resourcemanager.ResourceManager (LogAdapter.java:info(49)) - STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting ResourceManager
STARTUP_MSG: host = node-lambda-h01/129.0.30.180
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.1.1.3.1.4.0-315
STARTUP_MSG: classpath = /usr/hdp/3.1.4.0-315/hadoop/conf:/usr/hdp/3.1.4.0-315/hadoop/lib/jetty-security-9.3.24.v20180605.jar:/usr/hdp/3
.1.4.0-315/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/3.1.4.0-315/hadoop-yarn/.//hadoop-yarn-server-router.jar:/usr/hdp/3.1.4.0-315/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-3.1.1.3.1.4.0-315.jar:/usr/hdp/3.1.4.0-315/hadoop-yarn/.//hadoop-yarn-server-web-proxy-3.1.1.3.1.4.0-315.jar
:/usr/hdp/3.1.4.0-315/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/hdp/3.1.4.0-315/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/3.1.4.0-315/hadoop-yarn/.//hadoop-yarn-common-3.1.1.3.1.4.0-315.jar
:/usr/hdp/3.1.4.0-315/hadoop-yarn/.//hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/3.1.4.0-315/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:
-------------------------------------------------------------------------------------------------- other jar library
STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 58d0fd3d8ce58b10149da3c717c45e5e57a60d14; compiled by 'jenkins' on 2019-08-23T05:15Z
STARTUP_MSG: java = 1.8.0_112
************************************************************/
2020-04-03 17:33:17,600 INFO resourcemanager.ResourceManager (LogAdapter.java:info(49)) - registered UNIX signal handlers for [TERM, HUP, INT]
2020-04-03 17:33:17,779 FATAL resourcemanager.ResourceManager (ResourceManager.java:main(1516)) - Error starting ResourceManager
java.lang.NoSuchMethodError: org.apache.hadoop.yarn.util.SystemClock.getInstance()Lorg/apache/hadoop/yarn/util/SystemClock;
at org.apache.hadoop.yarn.server.resourcemanager.RMActiveServiceContext.<init>(RMActiveServiceContext.java:104)
at org.apache.hadoop.yarn.server.resourcemanager.RMContextImpl.<init>(RMContextImpl.java:104)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:251)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1512)
2020-04-03 17:33:17,785 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service ResourceManager failed in state STOPPED
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1296)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1358)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67)
at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2020-04-03 17:33:17,786 WARN service.AbstractService (ServiceOperations.java:stopQuietly(104)) - When stopping the service ResourceManager
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1296)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1358)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67)
at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2020-04-03 17:33:17,787 INFO resourcemanager.ResourceManager (LogAdapter.java:info(49)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at node-lambda-h01/129.0.30.180
************************************************************/

1 REPLY 1
Highlighted

Re: HDP 3.1.1 YARN Resource manager starting fail

New Contributor

Hi @KuldeepK @chennuri_gauris @Shelton, 

 

This is just a gentle reminder. I really need your help in looking into the below issue.

 

It has been almost a week since I am stuck at the same issue.  I would appreciate any quick help on this critical issue, which is blocking my tasks. 

 

I am a newbie to Cloudera Community and have seen that both of you are Gurus.  Can you please help me in fixing this issue? 

 

Thanks and Regards,

 

Horace

Don't have an account?
Coming from Hortonworks? Activate your account here