https://community.cloudera.com/t5/Support-Questions/DataNode-Data-Directory-is-getting-Full-on-one-particular/m-p/287637#M213157 <P><a href="https://community.cloudera.com/t5/user/viewprofilepage/user-id/73232">@nk_11</a>&nbsp;<BR /><BR /></P><P>HDFS data might not always be distributed uniformly across DataNodes. If the DataNodes are not balancing the data properly then you can run the HDFS Balancer from ambari UI.</P><LI-CODE lang="markup">Ambari UI --&gt; HDFS --&gt; "Service Actions" (Drop Down) --&gt; Rebalance HDFS </LI-CODE><P><BR /><A href="https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/managing-and-monitoring-ambari/content/amb_rebalance_hdfs_blocks.html" target="_blank">https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/managing-and-monitoring-ambari/content/amb_rebalance_hdfs_blocks.html</A></P><P>&nbsp;</P><P><BR />As you mentioned that you are also getting <STRONG><EM>"node manager unhealthy alert has the threshold limit is set to 90%"</EM></STRONG></P><P>Do you mean the "NodeManager is unhealthy" alert due to local-dirs (or local-dirs are bad errors)?</P><P>If yes, then it may be because the "<EM><STRONG>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</STRONG></EM>" property of YARN config is by default set to 90%.</P><P>If the utilization in the yarn disk (in this case /data) is above the limit set by yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage, try these options:</P><P>1. Free up some disk space<BR />(OR)<BR />2. Try to increase the value for "yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage" through Ambari. Followed by NodeManager restart.</P><P>&nbsp;</P> Tue, 14 Jan 2020 20:13:49 GMT jsensharma 2020-01-14T20:13:49Z