Hadoop Balancer

[#1]

Hi ,

How to run Hadoop Balancer on a single node, in our present environment one of the data node is not balanced (disk usage at 99%) remaining all data nodes are at 55%. We are using HDP version(2.3.4)

We want to balance only that single node which is at 99%.

Thanks in Advance.

[#2 · Accepted solution]

are you asking about balancing HDFS so that all nodes are evenly distributed? In that case you need to run HDFS balancer and it will spread that node's data across nodes, otherwise it defeats the point of balancing out HDFS. You can use -include -f hostsfile to hint which nodes to run balancer against but idea is you need more than one node there, is meant to be across multiple datanodes not single datanode.

https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#balancer

If your question is to balance data across all disks on one node, you can use disk balancer, sadly it's a new feature in Hadoop 3. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html

[#3]

Hi,

We ran the HDFS balancer on one of the node in our environment but it fails to run on single node rather it started on entire cluster. 😞

We ran the balancer on single node by adding "-source -f HOSTS.TXT" to the command

Thanks.

[#4]

balancer is not for single node. It is for balancing load on the cluster. For balancing load among different disks on the same node, a new disk balancer will be available in Hadoop version 3.0 as the link Artem shared, shows. There is not much you would be able to do here, except for the fact that "don't worry, Hadoop is smart enough to know that particular disk doesn't have any more space and it will find another disk" 🙂

[#5]

@Vijay

Hdfs balancer run across all the datanodes in appropriate distributing of the blocks, and the hadoop have the intelligent to shuffle the blocks in organized manner. so the balancer is not limited to sing node, you can just include and exclude the node for balancing. if you initiate the balancer in one node by excluding all but one, then its just like organizing the block on same node , so your disk utility will never reduce.

In your case just run the balancer accross the cluster, it will reduce the disk utility on the node, which is 99% filled.

[#6]

You also use -include -f with more than one host, not just that single datanode, I thought I was clear on that.

[#7]

Thank you guys @mqureshi @Rajendra M @Artem Ervits

[#8]

absolutely, happy to help.

[#9]

welcome,Thank you.

[#10]

In order to run a balancer on only one data node

hdfs balancer -include -f <#datanode name can be specified>

this would balance the data load on that particular DN.