Created 09-20-2017 09:51 AM
We have 32 datanodes in the cluster and recently added 2 new datanodes in cluster. However, the data replication after running the load balancer is very slow and takes a lot of time. Does modifying the parameter dfs.datanode.max.transfer.threads have an impact on this? Also, how to calculate the value to which it should be set.
Created 09-20-2017 08:01 PM
Here are the steps to increase HDFS Balancer network bandwidth for faster balancing of data between nodes
hdfs dfsadmin -setBalancerBandwidth 100000000
on all the DN and the client we ran the command below
hdfs balancer -Dfs.defaultFS=hdfs://<NN_HOSTNAME>:8020 -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.max.concurrent.moves=5 -Ddfs.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 5
This will faster balance your HDFS data between datanodes and do this when the cluster is not heavily used.
Couple of links to article : https://community.hortonworks.com/articles/51935/how-to-increase-hdfs-balancer-network-bandwidth-fo....
https://community.hortonworks.com/articles/43849/hdfs-balancer-2-configurations-cli-options.html
Hope this helps you.
Created 09-20-2017 08:01 PM
Here are the steps to increase HDFS Balancer network bandwidth for faster balancing of data between nodes
hdfs dfsadmin -setBalancerBandwidth 100000000
on all the DN and the client we ran the command below
hdfs balancer -Dfs.defaultFS=hdfs://<NN_HOSTNAME>:8020 -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.max.concurrent.moves=5 -Ddfs.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 5
This will faster balance your HDFS data between datanodes and do this when the cluster is not heavily used.
Couple of links to article : https://community.hortonworks.com/articles/51935/how-to-increase-hdfs-balancer-network-bandwidth-fo....
https://community.hortonworks.com/articles/43849/hdfs-balancer-2-configurations-cli-options.html
Hope this helps you.