Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDFS DataNode Data Directory Monitoring

HDFS DataNode Data Directory Monitoring

Good day.   We have an environment that was originally undersized with a single JBOD mount of 1TB.  When it filled up, we went ahead and added four additional 1TB mounts on each host.

 

The behavior in CM we are getting is that the HDFS 'DataNode Data Directory' is red because of the single mount being past the threshold.  We don't think rebalancing will work as we understand that to be at the data node level, rather than at the volume level.  At least that seems to be the behavior the last time we tried it.

 

Is there a way to possibly adjust this, or modify the monitor to look at the aggregate storage rather than a single volume?

 

The following DataNode Data Directory are on filesystems with less than 5.0 GiB of their space free. /cloudera/data/04/dfs/dn (free: 2.4 GiB (0.24%), capacity: 1,007.8 GiB) This role's DataNode Data Directory (/cloudera/data/02/dfs/dn, /cloudera/data/05/dfs/dn, /cloudera/data/1/dfs/dn, /cloudera/data/03/dfs/dn) are on a filesystem with more than 10.0 GiB of its space free

 

 

/dev/sdc1 1008G 459G 499G 48% /cloudera/data/1
/dev/sdb1 1008G 454G 503G 48% /cloudera/data/02
/dev/sde1 1008G 448G 509G 47% /cloudera/data/03
/dev/sdf1 1008G 956G 1.3G 100% /cloudera/data/04
/dev/sdg1 1008G 448G 509G 47% /cloudera/data/05