Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎03-27-2019

Log Directories Free Space

Bad : The following NodeManager Local Directories are on filesystems with less than 5.0 GiB of their space free. /data/08/yarn/nm (free: 33.2 MiB (0.00%), capacity: 931.1 GiB), /data/22/yarn/nm (free: 959.7 MiB (0.10%), capacity: 931.1 GiB), /data/17/yarn/nm (free: 1.3 GiB (0.14%), capacity: 931.1 GiB), /data/16/yarn/nm (free: 2.2 GiB (0.24%), capacity: 931.1 GiB), /data/13/yarn/nm (free: 113.4 MiB (0.01%), capacity: 931.1 GiB), /data/14/yarn/nm (free: 115.0 MiB (0.01%), capacity: 931.1 GiB), /data/05/yarn/nm (free: 103.1 MiB (0.01%), capacity: 931.1 GiB) This role's NodeManager Local Directories (/data/15/yarn/nm, /data/09/yarn/nm, /data/20/yarn/nm, /data/01/yarn/nm, /data/07/yarn/nm, /data/02/yarn/nm, /data/10/yarn/nm, /data/04/yarn/nm, /data/11/yarn/nm, /data/12/yarn/nm, /data/03/yarn/nm, /data/06/yarn/nm, /data/18/yarn/nm, /data/21/yarn/nm, /data/19/yarn/nm) are on a filesystem with more than 10.0 GiB of its space free.

 

 

 

Change NodeManager Local Directories Free Space Monitoring Absolute Thresholds for all roles in NodeManager Default Group role group
Change NodeManager Local Directories Free Space Monitoring Percentage Thresholds for all roles in NodeManager Default Group role group
Change NodeManager Local Directories Free Space Monitoring Absolute Thresholds for this role instance
Change NodeManager Local Directories Free Space Monitoring Percentage Thresholds for this role instance
View the log for this role instance at the time of the health test

 

 

 


Advice
This is a NodeManager health test that checks that the filesystem containing the NodeManager Local Directories of this NodeManager has sufficient free space. See the NodeManager Local Directories description on the NodeManager configuration page for more information on this directory type.

This test can be configured using the NodeManager Local Directories Free Space Monitoring Absolute Thresholds and NodeManager Local Directories Free Space Monitoring Percentage Thresholds NodeManager monitoring settings.

 

 

 

Posts: 1,903
Kudos: 435
Solutions: 305
Registered: ‎07-31-2013

Re: Log Directories Free Space

I am not seeing a question in your description posted here, so I assume your ask is how to get rid of this alert?

The alert exists to tell that you may face job failures due to exhaustion of available transient storage space in YARN.

Typically HDFS and YARN worker roles share disks between them. If you're observing this alert, it could be one of these hogging up most of the space, leaving little for the other. For HDFS you can check the DataNodes usage reports on the NameNode Web UI or via du calls run over the HDFS DN dirs on the reported host(s). A stale snapshot is often the cause of excessive, unaccounted space use by HDFS.

For YARN, typically the space is cleared soon after the container using it ends, but if you face this even when there are no jobs running, try investigating the contents of the directories shown in the alert.