Hadoop balancer performing very slow in our production cluster.
CDH version 5.5.1 express edition
Cluster capacity 850 nodes.
Highest % occupies nodes - 85%
Lowest % occupied nodes - 5 %
Recently we have added 40 nodes in our production cluster and initiated the hadoop balancer with following parameters but there is no significant improvement while transfering the data Can you please suggest what are parameters which I need to configure to improve the performance.
Is it possible to transfer TB of data in each iteation.
sudo -u apps hdfs balancer
-Ddfs.balancer.movedWinWidth=5400000 \
-Ddfs.balancer.moverThreads=2000 \
-Ddfs.balancer.dispatcherThreads=2000 \
-Ddfs.balance.bandwidthPerSec=200000000 \
-Ddfs.balancer.getBlocks.size=1000000000000 \
-Ddfs.datanode.balance.max.concurrent.moves=24 \
-Ddfs.balancer.getBlocks.min-block-size=536870912 -threshold 15 -include <Datanodes>
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
Jan 18, 2019 7:44:55 AM 61 568.44 GB 136.22 TB 280 GB
Jan 18, 2019 7:52:39 AM 62 576.80 GB 136.22 TB 280 GB
Jan 18, 2019 7:58:23 AM 63 583.06 GB 136.21 TB 280 GB
Jan 18, 2019 8:04:36 AM 64 590.99 GB 136.21 TB 280 GB
Jan 18, 2019 8:10:48 AM 65 602.05 GB 136.24 TB 280 GB
