Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDFS balancing , how to balanced hdfs data ?

HDFS balancing , how to balanced hdfs data ?

Hi all


dear friends and colleges , and hortonworks expert

please help me with the following isshue

this issue is happen on all our ambari clusters , and we have many


please let me know if this is a BUG or maybe something that we can fix?




we have HDP version - 2.6.4


On the datanode machine we can see that hdfs data isn’t balanced

On some disks we have different size as sdb 11G and sdd 17G


/dev/sdd 20G 3.0G 17G 15% /grid/sdd 
/dev/sdb 20G 11G 9.3G 53% /grid/sdb <-- WHY DISK DISK NOT BALANCED AS SDD DISK , WHY DISKS ARE DIFF USED SIZE!!!


After searching in google I found the following CLI ( from https://community.hortonworks.com/questions/19694/help-with-exception-from-hdfs-balancer.html )


hdfs balancer -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 20 1>/tmp/balancer-out.log 2>/tmp/balancer-debug.log



and after I run it we get the same hdfs size


/dev/sdd 20G 3.0G 17G 15% /grid/sdd
/dev/sdb 20G 11G 9.3G 53% /grid/sdb


more /tmp/balancer-out.log Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved The cluster is balanced. Exiting... Mar 7, 2019 5:02:34 PM 0 0 B 0 B 0 B Mar 7, 2019 5:02:34 PM Balancing took 1.453 seconds


So actually we not get balanced in hdfs


Please advise , how to balance the hdfs data so all disk will be with the same used size

Michael-Bronson