Unable to start YARN - NodeManager on role NodeManager

[#1]

Execute command Start this NodeManager on role NodeManager 

Failed to start role

Supervisor returned FATAL. Please check the role log file, stderr, or stdout.

Environment details:

Version: Cloudera Express 5.15.0
Java VM Name: Java HotSpot(TM) 64-Bit Server VM
Java VM Vendor: Oracle Corporation
Java Version: 1.7.0_67
 
System details:
Linux optim-rhel72-uppu.development.unicomglobal.software 3.10.0-327.28.3.el7.x86_64 #1 SMP Fri Aug 12 13:21:05 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

I have followed the steps under "Configuring TLS/SSL for HDFS, YARN and MapReduce" using the link https://www.cloudera.com/documentation/enterprise/5-15-x/topics/sg_hive_encryption.html
 
Service did not start successfully; not all of the required roles started: only 0/1 roles started. Reasons : Service has only 0 NodeManager roles running instead of minimum required 1

I see below error in the role log:

Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:269)
 at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562)
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)
Caused by: java.io.IOException: Linux container executor not configured properly (error=24)
 at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:199)
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:267)
 ... 3 more
Caused by: ExitCodeException exitCode=24: Invalid conf file provided : /etc/hadoop/conf.cloudera.yarn/container-executor.cfg

 at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
 at org.apache.hadoop.util.Shell.run(Shell.java:507)
 at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
 at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:193)
 ... 4 more

SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at optim-rhel72-uppu.development.unicomglobal.software/10.1.72.3
************************************************************/

Any help highly appreciated.

Thanks,

Tulasi

[#2]

Hi Tulasi,

Could you please verify the "container-executor.group" are same on both from Cloudera manager (Yarn->Configuration->Container Executor Group) and /etc/hadoop/conf.cloudera.yarn/container-executor.cfg (from Node manager host)

Let us know if you have questions

Thanks
Jerry

[#3]

Hi Jerry,

Here is what I have on my system:

/etc/hadoop/conf.cloudera.yarn

-r-------- 1 root hadoop  156 Jan 24 01:00 container-executor.cfg
-rw-r--r-- 1 root root   3894 Jan 17 22:56 core-site.xml
-rw-r--r-- 1 root root    617 Jan 17 22:56 hadoop-env.sh
-rw-r--r-- 1 root root   2729 Jan 17 22:56 hdfs-site.xml

Even if I change above file permission, after start, it changes back to the same permission.

From manager I have this

Container Executor Group = yarn

Upgrade also not allowing as it requires all services should be up and running.

Let me know if you need any more details.

Thanks,

Tulasi

[#4]

Hi Tulasi,

Could you check the value of this property Container Executor Group from the file "container-executor.cfg" file and cross check with CM configuration

Thanks

Jerry

[#5]

Hi Jerry, Yes, the value of "Container Executor Group" property is matching with CM, see below: [root@optim-rhel72-uppu conf.cloudera.yarn]# cat container-executor.cfg yarn.nodemanager.linux-container-executor.group=yarn min.user.id=1000 allowed.system.users=nobody,impala,hive,llama,hbase banned.users=hdfs,yarn,mapred,bin Thanks, Tulasi

[#6]

Hi @Tulasi, 

Could you please send us the output of below command on all the NodeManager hosts?

ls -alt /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor

The correct permission should be like this:

---Sr-s--- 1 root yarn 53728 Jan 28 14:03 /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor

If it looks different, you can perform the following steps on all NodeManagers:

chmod 6050  /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor
chgrp yarn /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor

Thanks and hope this helps,

Li

[#7]

Hi Li,

I changed the permission, still it didn't fix the problem.

[root@optim-rhel72-uppu bin]# ls -alt /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor
---Sr-s--- 1 root yarn 53712 May 24  2018 /opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/container-executor

In this below, is it something to do with banned.users? 

[root@optim-rhel72-uppu conf.cloudera.yarn]# cat /etc/hadoop/conf.cloudera.yarn/container-executor.cfg
yarn.nodemanager.linux-container-executor.group=yarn
min.user.id=1000
allowed.system.users=nobody,impala,hive,llama,hbase
banned.users=hdfs,yarn,mapred,bin

Thanks,

Tulasi

[#8]

Hi @Tulasi,

The banned.users property is to prevent jobs from being submitted using those user accounts. It should not cause NodeManager not able to start problem.

I suggest you checking these doc links:

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cdh_sg_other_hadoop_security.html#to...

and

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cdh_sg_yarn_container_exec_errors.ht...

How many nodes does your cluster have? Have you checked all the permissions?

Thanks,

Li

[#9]

Hi Li,

Everything on a single node.

/opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin
[root@optim-rhel72-uppu bin]# ls -lrt
total 80
-rwxr-xr-x 1 root root 12476 May 24  2018 yarn
-rwxr-xr-x 1 root root  5463 May 24  2018 mapred
---Sr-s--- 1 root yarn 53712 May 24  2018 container-executor

This is the error from /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-optim-rhel72-uppu.development.unicomglobal.software.log.out

2019-01-30 23:42:45,872 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:269)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)
Caused by: java.io.IOException: Cannot run program "/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-yarn/bin/container-executor": error=13, Permission denied

Thanks,

Tulasi

[#10]

Hi @Tulasi,

Could you please run below command and send us the output?

id yarn

Thanks,

Li

[#11]

Hi Li,

This is what I am getting:

[root@optim-rhel72-uppu ~]# id yarn
uid=1007(yarn) gid=1010(hadoop) groups=1010(hadoop)

Thanks,

Tulasi

[#12]

Is nosuid set on the mount point?  I had a simialr issue documented here: http://community.cloudera.com/t5/Cloudera-Manager-Installation/URGENT-Cluster-unavailable-after-upgr...

[#13]

This is the content of my /etc/fstab file

-----------------------------------

/dev/mapper/rhel_rhel72-root /                       xfs     defaults        0 0
UUID=d762b842-5c87-4e4d-bc0e-7a6bad357604 /boot                   xfs     defaults        0 0
/dev/mapper/rhel_rhel72-home /home                   xfs     defaults        0 0
/dev/mapper/rhel_rhel72-swap swap                    swap    defaults        0 0

 -----------------------------------

Do I need to change anything?

Thanks.

[#14 · Accepted solution]

Hi @Tulasi,

Sorry for my late reply. From the output you sent below:

[root@optim-rhel72-uppu ~]# id yarn
uid=1007(yarn) gid=1010(hadoop) groups=1010(hadoop)

This looks a little different than my test cluster. Can you please do this?

usermod -g yarn yarn
usermod -a -G hadoop yarn

Also, please paste the content of this file:

/opt/cloudera/parcels/CDH/meta/permissions.json

Thanks,

Li 

[#15]

Hi Li,

Thanks for being on top of it and helping me in solving the problem.

usermod -g yarn yarn

usermod -a -G hadoop yarn

Above two commands fixed my problem.

[root@optim-rhel72-uppu meta]# id yarn
uid=1007(yarn) gid=1008(yarn) groups=1008(yarn),1010(hadoop)

I have no idea how yarn user permissions are changed, all that I am following is that what have been suggested in cloudera instructions to enable encryption.

Thanks to all of the folks for providing suggestions.

Problems like this sucks lot of time in identifying where to fix and I would request cloudera to improve such situations.

Thanks,

Tulasi

[#16]

Hi @Tulasi,

Greate to hear the issue got resolved! I will report internally on this to our documentation team to see how we can improve on it.

Thanks,

Li

[#17]

Hi @Tulasi,

Try to kill running process on port used by Yarn Services and then Try to restart.

Regards,

Manu.

[#18]

Hi Manu, I tried your suggestion and it didn't work. Thanks, Tulasi