Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

[#1]

Error starting NodeManager

java.lang.NullPointerException

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverContainer(ContainerManagerImpl.java:289)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:252)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:235)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:250)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:445)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:492)

[#2]

@Pranshu Pranshu Need more information...Whats in the log files?

[#3]

@Neeraj Sabharwal Thanks for your assistance. I found the above details from the log at the instant when warning was generated. after that the service went down. Apart from this noting more was there in the log. Also the memory uses is just below the critical threshold capacity. We set 90 % for critical limit and right now it is 89 .4 %. So i mentioned is it related to memory issue.

[#4 · Accepted solution]

@Pranshu Pranshu

Login to datanodes or nodemanager node

cd /var/log/hadoop/hdfs

ls -lrt

look into the last log

for example: hadoop-hdfs-datanode-nodename.log

This log will give you the details. It can be related to the memory

[#5]

@Neeraj Sabharwal

got the logs.seems like It is related to memory issue. Unfortunately i don't have permissions to delete it. Can I create a soft link to move the data around as a work around. Can you please assist if I create a soft link for any lib, will it move the present data or the upcoming data or both ?

[#6]

@Pranshu Pranshu

Are you planning to delete logs? I don't suggest to create symlink without knowing the exact details.

Are you planning to move hdfs data?

[#7]

@Neeraj Sabharwal

Yes I was thinking to move the data, deletion is out of my boundaries as per my role.

[#8]

@Pranshu Pranshu Why do yo want to move the data? Is system running out of space?

[#9]

@Neeraj Sabharwal

yes the system is running out of space. Can you please suggest me a better way rather than creating a soft link.

[#10]

Is it prod?