How to make data replication faster after adding new nodes in cluster.

Piyali_Gupta — Wed, 20 Sep 2017 16:51:21 GMT

We have 32 datanodes in the cluster and recently added 2 new datanodes in cluster. However, the data replication after running the load balancer is very slow and takes a lot of time. Does modifying the parameter dfs.datanode.max.transfer.threads have an impact on this? Also, how to calculate the value to which it should be set.

Re: How to make data replication faster after adding new nodes in cluster.

bandarusridhar1 — Thu, 21 Sep 2017 03:01:59 GMT

@Piyali Gupta

Here are the steps to increase HDFS Balancer network bandwidth for faster balancing of data between nodes

Article

hdfs dfsadmin -setBalancerBandwidth 100000000

on all the DN and the client we ran the command below

hdfs balancer -Dfs.defaultFS=hdfs://<NN_HOSTNAME>:8020 -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.max.concurrent.moves=5 -Ddfs.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 5

This will faster balance your HDFS data between datanodes and do this when the cluster is not heavily used.

Couple of links to article : https://community.hortonworks.com/articles/51935/how-to-increase-hdfs-balancer-network-bandwidth-fo.html

https://community.hortonworks.com/articles/43849/hdfs-balancer-2-configurations-cli-options.html

Hope this helps you.

question Re: How to make data replication faster after adding new nodes in cluster. in Archives of Support Questions (Read Only)

How to make data replication faster after adding new nodes in cluster.

Re: How to make data replication faster after adding new nodes in cluster.

Article