hdfs rebalancing

[#1]

We are having a 10 dat node cluster. But data in hdfs is not spread across the nodes. some nodes are under utilized. Will a hdfs rebalancing activity resolve this issue. And During rebalancing will we be able to use the cluster.

[#2 · Accepted solution]

@ARUN

The whole purpose of “balancer” utility is to help balance the blocks across DataNodes in the cluster. So it should do the job, if there is no major issue at the cluster level.

It is usually recommend to run the balancer periodically during times when the cluster load is expected to be lower than usual.

Also please refer to the following article that explains the importance of balancer and the performance improvement facts: https://community.hortonworks.com/articles/43615/hdfs-balancer-1-100x-performance-improvement.html

You can run the HDFS balancer in Maintenance window as well as Without a Maintenance window. Few things you should keep in mind while running the balancer as mentioned in : https://community.hortonworks.com/articles/43849/hdfs-balancer-2-configurations-cli-options.html