Support Questions

Find answers, ask questions, and share your expertise

Nodemanager fails to start

I am not able to restart nodemanager , please find errors in the log :

Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/000002.dbtmp: No space left on device
        at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(
        at org.apache.hadoop.service.AbstractService.init(
        ... 5 more
2018-07-12 09:52:18,186 INFO  nodemanager.NodeManager ( - SHUTDOWN_MSG:

however there is space in the device , I removed yarn-nm-state to /tmp and restarted the nodemanager but other two nodemanager started but still one of the nodemanager is not starting .


Super Mentor

@Anurag Mishra

The error suggests that your host is sufferring from Disk Space Issue.

IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/000002.dbtmp: No space left on device

So you will need to make sure that the mentioned partition has enough disk free.

Can you please share the output of the following command?

# df -h 
# du -csh /var


@Jay Kumar SenSharma

/dev/mapper/vg1-logvol 60G 37G 20G 66% /var/log

Super Mentor

@Anurag Mishra

If you are noticing "No space left on device" in some cloud environment / OpenStack ..etc or on some VM then i will suggest rebooting the host VM once.

Because some times even if the VM has enough space (based on df -h output) still the VM is not allocate space for the apps

Please try rebooting the host and then you might not see the same error.