We have a 700 plus node setup with Rack awareness set. The total number of racks are ~40 (each rack name set to that of the physical rack) with a replication factor of 3. I am keen to find out what observations other users have had with respect to the HDFS balancer. Our setup has a number of under and over utilized nodes (Mean 80/Min 20/Max 90) and the balancer keeps on running forever. A number of params have been used to tune the balancer but no improvements have been observed.
Command sample : hdfs balancer -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=20000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.balance.bandwidthPerSec=200000000 -Ddfs.balancer.getBlocks.size=1000000000000 -Ddfs.datanode.balance.max.concurrent.moves=32 -Ddfs.balancer.getBlocks.min-block-size=536870912 -threshold 20
In my observation, the bytes being moved does not neccesarily reflect the progress, also the bytes left to move sometimes increases rather than going down.
Env : CDH 5.5.1 (Centos 6)
Data size : ~30 PB
Hi @Fawze. Your line of thought is spot on. Looking at our configs it does call for some changes to be made.
The details are as follows;
Datanode volume choosing policy = roundRobin ( Reckon available space would be best?)
Do all nodes have same storagre - Actually we have 3 HDD templates, some have 8, some 10 and some have 12 disks with the capacity of each individual HDD at ~10 TB.
( This might be throwing off the balancer. I wonder whether the Hadoop framework looks at individual disk capacity for balancing rather than the node as a whole unit of storage?)
Nodes under 20% = Approx 100 ( They've been around that even after running the balancer continuously for weeks)
Here's the node distribution -
Rack name nodes
Let me know you thoughts and thank you so much for chipping in :)
Sounds good @Fawze. We're planning on upgrading to CDH 5.16.1 (thank you for your views on that one too :) and I will check with out DC team if they can balance the racks in order to reflect the same number of hosts and I'll move the the reccommended policy.
Thank you so much!