Failed to initialize new NodeManager with ClouderaManager 5.15.0

[#1]

Failed to initialize new NodeManager with ClouderaManager 5.15.0.

After adding the role instance to a new server, we've got a problem when trying to start this NodeManager.

The following error appears:

2018-07-04 11:24:18,873 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container executor initialization is : 24
ExitCodeException exitCode=24: File /opt/cloudera must be owned by root, but is owned by 496

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
        at org.apache.hadoop.util.Shell.run(Shell.java:504)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:192)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:249)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
2018-07-04 11:24:18,884 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
2018-07-04 11:24:18,884 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:251)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
Caused by: java.io.IOException: Linux container executor not configured properly (error=24)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:198)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:249)
        ... 3 more
Caused by: ExitCodeException exitCode=24: File /opt/cloudera must be owned by root, but is owned by 496

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
        at org.apache.hadoop.util.Shell.run(Shell.java:504)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:192)
        ... 4 more
2018-07-04 11:24:18,888 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:251)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
Caused by: java.io.IOException: Linux container executor not configured properly (error=24)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:198)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:249)
        ... 3 more
Caused by: ExitCodeException exitCode=24: File /opt/cloudera must be owned by root, but is owned by 496

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
        at org.apache.hadoop.util.Shell.run(Shell.java:504)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:192)
        ... 4 more
2018-07-04 11:24:18,892 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:

Our CM was upgraded from 5.11.0 to 5.15.0 few days ago and this is the first time to initialize a new NodeManager with 5.15.0 (It's never happened in CM 5.11.0)

My current workaround is:

1. Change /opt/cloudera to owned by root
2. Then start NodeManager using CM
3. Change the owner back to cloudera-scm
4. Issue a restart on this NodeManager to make sure everything works all right

Does anyone have a similar problem? Any help will be greatly appreciated.

[#2]

There was a known issue in Cloudera Manager up until 5.14.4 and 5.15.1 that can cause the necessary container executor configuration files to not be deployed on new or recommissioned NodeManager hosts. The fix for this is noted at https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_rn_fixed_issues.html#OPSAP...

A better workaround than changing ownership (which is a red herring because of missing symlinks causing the executor binary to look at the wrong path) is to simply add a YARN Gateway role to all NodeManager hosts and perform a 'Deploy Client Configuration' under YARN.

Upgrading to 5.15.1 or higher when it comes out should help resolve this issue.