Reply
Explorer
Posts: 20
Registered: ‎12-27-2018
Accepted Solution

Unable to start YARN - NodeManager on role NodeManager

[ Edited ]

 

Execute command Start this NodeManager on role NodeManager 

Failed to start role

Supervisor returned FATAL. Please check the role log file, stderr, or stdout.

 

Environment details:

 

Version: Cloudera Express 5.15.0
Java VM Name: Java HotSpot(TM) 64-Bit Server VM
Java VM Vendor: Oracle Corporation
Java Version: 1.7.0_67
 
System details:
Linux optim-rhel72-uppu.development.unicomglobal.software 3.10.0-327.28.3.el7.x86_64 #1 SMP Fri Aug 12 13:21:05 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

 

I have followed the steps under "Configuring TLS/SSL for HDFS, YARN and MapReduce" using the link https://www.cloudera.com/documentation/enterprise/5-15-x/topics/sg_hive_encryption.html
 
Service did not start successfully; not all of the required roles started: only 0/1 roles started. Reasons : Service has only 0 NodeManager roles running instead of minimum required 1

I see below error in the role log:

 

Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:269)
 at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562)
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)
Caused by: java.io.IOException: Linux container executor not configured properly (error=24)
 at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:199)
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:267)
 ... 3 more
Caused by: ExitCodeException exitCode=24: Invalid conf file provided : /etc/hadoop/conf.cloudera.yarn/container-executor.cfg
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
 at org.apache.hadoop.util.Shell.run(Shell.java:507)
 at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
 at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:193)
 ... 4 more
 
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at optim-rhel72-uppu.development.unicomglobal.software/10.1.72.3
************************************************************/
 
Any help highly appreciated.
 
Thanks,
Tulasi
 

 

 

 

Posts: 45
Topics: 0
Kudos: 13
Solutions: 4
Registered: ‎07-30-2018

Re: Unable to start YARN - NodeManager on role NodeManager

Hi Tulasi,

Could you please verify the "container-executor.group" are same on both from Cloudera manager (Yarn->Configuration->Container Executor Group) and /etc/hadoop/conf.cloudera.yarn/container-executor.cfg (from Node manager host)

Let us know if you have questions

Thanks
Jerry

Explorer
Posts: 20
Registered: ‎12-27-2018

Re: Unable to start YARN - NodeManager on role NodeManager

Hi Jerry,

 

Here is what I have on my system:

 

/etc/hadoop/conf.cloudera.yarn

-r-------- 1 root hadoop  156 Jan 24 01:00 container-executor.cfg
-rw-r--r-- 1 root root   3894 Jan 17 22:56 core-site.xml
-rw-r--r-- 1 root root    617 Jan 17 22:56 hadoop-env.sh
-rw-r--r-- 1 root root   2729 Jan 17 22:56 hdfs-site.xml

 

Even if I change above file permission, after start, it changes back to the same permission.

 

From manager I have this

Container Executor Group = yarn

 

Upgrade also not allowing as it requires all services should be up and running.

 

Let me know if you need any more details.

 

Thanks,

Tulasi

 

Expert Contributor
Posts: 102
Registered: ‎02-23-2018

Re: Unable to start YARN - NodeManager on role NodeManager

Hi @Tulasi,

 

Try to kill running process on port used by Yarn Services and then Try to restart.

 

Regards,

Manu.

Posts: 45
Topics: 0
Kudos: 13
Solutions: 4
Registered: ‎07-30-2018

Re: Unable to start YARN - NodeManager on role NodeManager

Hi Tulasi,

 

Could you check the value of this property Container Executor Group from the file "container-executor.cfg" file and cross check with CM configuration

 

Thanks

Jerry

Explorer
Posts: 20
Registered: ‎12-27-2018

Re: Unable to start YARN - NodeManager on role NodeManager

Hi Manu, I tried your suggestion and it didn't work. Thanks, Tulasi
Explorer
Posts: 20
Registered: ‎12-27-2018

Re: Unable to start YARN - NodeManager on role NodeManager

Hi Jerry, Yes, the value of "Container Executor Group" property is matching with CM, see below: [root@optim-rhel72-uppu conf.cloudera.yarn]# cat container-executor.cfg yarn.nodemanager.linux-container-executor.group=yarn min.user.id=1000 allowed.system.users=nobody,impala,hive,llama,hbase banned.users=hdfs,yarn,mapred,bin Thanks, Tulasi
Highlighted
Posts: 933
Kudos: 28
Solutions: 16
Registered: ‎05-27-2014

Re: Unable to start YARN - NodeManager on role NodeManager

Hi @Tulasi

 

Could you please send us the output of below command on all the NodeManager hosts?

 

ls -alt /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor

 

The correct permission should be like this:

---Sr-s--- 1 root yarn 53728 Jan 28 14:03 /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor

 

If it looks different, you can perform the following steps on all NodeManagers:

chmod 6050  /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor
chgrp yarn /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor

Thanks and hope this helps,

Li

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Explorer
Posts: 20
Registered: ‎12-27-2018

Re: Unable to start YARN - NodeManager on role NodeManager

Hi Li,

 

I changed the permission, still it didn't fix the problem.

 

[root@optim-rhel72-uppu bin]# ls -alt /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor
---Sr-s--- 1 root yarn 53712 May 24  2018 /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor

 

In this below, is it something to do with banned.users? 

 

[root@optim-rhel72-uppu conf.cloudera.yarn]# cat /etc/hadoop/conf.cloudera.yarn/container-executor.cfg
yarn.nodemanager.linux-container-executor.group=yarn
min.user.id=1000
allowed.system.users=nobody,impala,hive,llama,hbase
banned.users=hdfs,yarn,mapred,bin

 

Thanks,

Tulasi

Posts: 933
Kudos: 28
Solutions: 16
Registered: ‎05-27-2014

Re: Unable to start YARN - NodeManager on role NodeManager

Hi @Tulasi,

 

The banned.users property is to prevent jobs from being submitted using those user accounts. It should not cause NodeManager not able to start problem.

 

I suggest you checking these doc links:

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cdh_sg_other_hadoop_security.html#to...

and

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cdh_sg_yarn_container_exec_errors.ht...

 

How many nodes does your cluster have? Have you checked all the permissions?

 

Thanks,

Li

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Announcements