question Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ? in Support Questions

Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

sinha_pranshu — Tue, 16 Feb 2016 06:50:33 GMT

Error starting NodeManager

java.lang.NullPointerException

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverContainer(ContainerManagerImpl.java:289)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:252)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:235)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:250)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:445)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:492)

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

nsabharwal — Tue, 16 Feb 2016 07:02:36 GMT

@Pranshu Pranshu Need more information...Whats in the log files?

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

sinha_pranshu — Tue, 16 Feb 2016 07:26:54 GMT

@Neeraj Sabharwal Thanks for your assistance. I found the above details from the log at the instant when warning was generated. after that the service went down. Apart from this noting more was there in the log. Also the memory uses is just below the critical threshold capacity. We set 90 % for critical limit and right now it is 89 .4 %. So i mentioned is it related to memory issue.

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

nsabharwal — Tue, 16 Feb 2016 07:31:47 GMT

@Pranshu Pranshu

cd /var/log/hadoop/hdfs

ls -lrt

look into the last log

for example: hadoop-hdfs-datanode-nodename.log

This log will give you the details. It can be related to the memory

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

sinha_pranshu — Tue, 16 Feb 2016 07:55:11 GMT

@Neeraj Sabharwal

got the logs.seems like It is related to memory issue. Unfortunately i don't have permissions to delete it. Can I create a soft link to move the data around as a work around. Can you please assist if I create a soft link for any lib, will it move the present data or the upcoming data or both ?

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

nsabharwal — Tue, 16 Feb 2016 08:30:40 GMT

@Pranshu Pranshu

Are you planning to delete logs? I don't suggest to create symlink without knowing the exact details.

Are you planning to move hdfs data?

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

sinha_pranshu — Tue, 16 Feb 2016 09:09:03 GMT

@Neeraj Sabharwal

Yes I was thinking to move the data, deletion is out of my boundaries as per my role.

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

nsabharwal — Tue, 16 Feb 2016 09:12:28 GMT

@Pranshu Pranshu Why do yo want to move the data? Is system running out of space?

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

sinha_pranshu — Wed, 17 Feb 2016 05:53:56 GMT

@Neeraj Sabharwal

yes the system is running out of space. Can you please suggest me a better way rather than creating a soft link.

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

nsabharwal — Wed, 17 Feb 2016 06:08:44 GMT

Is it prod?

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

sinha_pranshu — Wed, 17 Feb 2016 06:38:34 GMT

Yes. it is prod environment.

Re: Yarn went down on my cluster. I restarted the service but it didn't work. Does it have any thing to do with housekeeping on Cluster ?

nsabharwal — Wed, 17 Feb 2016 06:41:13 GMT

I don't recommend doing anything like this in prod .

Please open a support ticket and have someone from support to look into this.