Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Node manager keepts exiting

Node manager keepts exiting

Expert Contributor

I have a 4 Node cluster. On my Master Node, Node Manager is continuously exiting. It gets restarted on it own but It's disturbing some of my processes. For this reason, I have stopped Node Manager on Master Node.

Of course that solves my problem with processes getting interrupted because of Node Manager restart but Under Cluster Node metrics, It shows me 1 Lost node (which makes sense, There is not Node Manager running on this node).

I have also tried increasing Heap Memory for NameNode and Secondary Name Node but that did not help.

Please suggest what can be done to fix this?

In Hadoop yarn folder logs,

Apr 17, 3:03:05.946 PMINFOorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutorDeleting path : /yarn/container-logs/application_1523905807460_4113/container_1523905807460_4113_01_000001/stderr
Apr 17, 3:03:05.963 PMINFOorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutorDeleting path : /yarn/container-logs/application_1523905807460_4113
Apr 17, 3:03:07.091 PMWARNorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutorExit code from container container_1523905807460_4129_01_000001 is : 137
Apr 17, 3:03:07.091 PMINFOorg.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerContainer container_1523905807460_4129_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE
Apr 17, 3:03:07.091 PMINFOorg.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunchCleaning up container container_1523905807460_4129_01_000001
Apr 17, 3:03:07.111 PMINFOorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutorDeleting absolute path : /yarn/nm/usercache/hue/appcache/application_1523905807460_4129/container_1523905807460_4129_01_000001
Apr 17, 3:03:07.112 PMWARNorg.apache.hadoop.yarn.server.nodemanager.NMAuditLoggerUSER=hue	OPERATION=Container Finished - Failed	TARGET=ContainerImpl	RESULT=FAILURE	DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE	APPID=application_1523905807460_4129	CONTAINERID=container_1523905807460_4129_01_000001
Apr 17, 3:03:07.112 PMINFOorg.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerContainer container_1523905807460_4129_01_000001 transitioned from EXITED_WITH_FAILURE to DONE
Apr 17, 3:03:07.112 PMINFOorg.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationRemoving container_1523905807460_4129_01_000001 from application application_1523905807460_4129
Apr 17, 3:03:07.112 PMINFOorg.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImplConsidering container container_1523905807460_4129_01_000001 for log-aggregation
Apr 17, 3:03:07.112 PMINFOorg.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesGot event CONTAINER_STOP for appId application_1523905807460_4129


Re: Node manager keepts exiting


Do you have disk contention? We have seen issues where multiple services running on same node all trying to write to same physical disk, end up contending with each other.

We'd had to move services to write to different disks in the machines in those cases.

Don't have an account?
Coming from Hortonworks? Activate your account here