Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Failed to initialize new NodeManager with ClouderaManager 5.15.0

Highlighted

Failed to initialize new NodeManager with ClouderaManager 5.15.0

New Contributor

Failed to initialize new NodeManager with ClouderaManager 5.15.0.

After adding the role instance to a new server, we've got a problem when trying to start this NodeManager.

 

The following error appears:

 

2018-07-04 11:24:18,873 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container executor initialization is : 24
ExitCodeException exitCode=24: File /opt/cloudera must be owned by root, but is owned by 496

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
        at org.apache.hadoop.util.Shell.run(Shell.java:504)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:192)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:249)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
2018-07-04 11:24:18,884 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
2018-07-04 11:24:18,884 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:251)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
Caused by: java.io.IOException: Linux container executor not configured properly (error=24)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:198)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:249)
        ... 3 more
Caused by: ExitCodeException exitCode=24: File /opt/cloudera must be owned by root, but is owned by 496

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
        at org.apache.hadoop.util.Shell.run(Shell.java:504)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:192)
        ... 4 more
2018-07-04 11:24:18,888 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:251)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:544)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:591)
Caused by: java.io.IOException: Linux container executor not configured properly (error=24)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:198)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:249)
        ... 3 more
Caused by: ExitCodeException exitCode=24: File /opt/cloudera must be owned by root, but is owned by 496

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
        at org.apache.hadoop.util.Shell.run(Shell.java:504)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:192)
        ... 4 more
2018-07-04 11:24:18,892 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:

Our CM was upgraded from 5.11.0 to 5.15.0 few days ago and this is the first time to initialize a new NodeManager with 5.15.0 (It's never happened in CM 5.11.0)

 

My current workaround is:

  1. Change /opt/cloudera to owned by root
  2. Then start NodeManager using CM
  3. Change the owner back to cloudera-scm
  4. Issue a restart on this NodeManager to make sure everything works all right

Does anyone have a similar problem? Any help will be greatly appreciated.

1 REPLY 1

Re: Failed to initialize new NodeManager with ClouderaManager 5.15.0

Master Guru
There was a known issue in Cloudera Manager up until 5.14.4 and 5.15.1 that can cause the necessary container executor configuration files to not be deployed on new or recommissioned NodeManager hosts. The fix for this is noted at https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_rn_fixed_issues.html#OPSAP...

A better workaround than changing ownership (which is a red herring because of missing symlinks causing the executor binary to look at the wrong path) is to simply add a YARN Gateway role to all NodeManager hosts and perform a 'Deploy Client Configuration' under YARN.

Upgrading to 5.15.1 or higher when it comes out should help resolve this issue.