One disk of the datanode is at 99% used...

[#1]

Hi,

I Have a hadoop cluster 3.0.1 with 3 journalnodes, 1 nfsgateways node and 6 workernodes. I connected by ssh to the worker nodes today and realised by doing a "df -h" that one a the one local disk (/data/4) is around 94% used on every worker nodes whereas the others disk are between 50% and 65%...  

The HDFS status on the another hand is the following:

Disk Usage (DFS Used)          44.77%   28.1 TB / 62.8 TB

Disk Usage (Non DFS Used)  14.97% 9.4 TB / 62.8 TB

Disk Remaining                       40.26%  25.3 TB / 62.8 TB

What are the the elements i should check to make sure that a full local disk won't create any issue?

[#2 · Accepted solution]

@Koffi 

If your DataNodes are unevenly distributed/loaded then HDFS provides an option to Balance them using the "HDFS Balancer" utility.

HDFS balancer utility helps to balance the blocks across DataNodes in the cluster. 

Via Ambari: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/managing-and-monitoring-ambari/content/amb_reb...

Further Details: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/data-storage/content/balancer_commands.html

.