The namenode in our Hortonworks cluster just shut down on its own last night, with the secondary namenode taking over. When investigating the issue this morning, I can't find any errors in the namenode's log before the shutdown but it does flood its log with the following warning:
WARN namenode.NameNodeResourceChecker (NameNodeResourceChecker.java:isResourceAvailable(89)) - Space available on volume '/dev/mapper/vg00-lvopt' is 0, which is below the configured reserved amount 104857600
Yet there seems to be plenty of space there:
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-lvopt 29G 2.5G 27G 9% /opt
I figured it might be a permissions problem, since ambari is running as non-root, yet its sudoers configuration on the namenode machine is IDENTICAL to the sudoers config on the secondary namenode machine (whose logs look fine, no warning or errors there). Anyone has any ideas what's causing this and how to solve it?