question Re: Hadoop Balancer in Archives of Support Questions (Read Only)

Hadoop Balancer

reddyr211 — Wed, 08 Feb 2017 10:31:43 GMT

Hi ,

How to run Hadoop Balancer on a single node, in our present environment one of the data node is not balanced (disk usage at 99%) remaining all data nodes are at 55%. We are using HDP version(2.3.4)

We want to balance only that single node which is at 99%.

Thanks in Advance.

Re: Hadoop Balancer

aervits — Wed, 08 Feb 2017 11:13:28 GMT

are you asking about balancing HDFS so that all nodes are evenly distributed? In that case you need to run HDFS balancer and it will spread that node's data across nodes, otherwise it defeats the point of balancing out HDFS. You can use -include -f hostsfile to hint which nodes to run balancer against but idea is you need more than one node there, is meant to be across multiple datanodes not single datanode.

https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#balancer

If your question is to balance data across all disks on one node, you can use disk balancer, sadly it's a new feature in Hadoop 3. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html

Re: Hadoop Balancer

reddyr211 — Wed, 08 Feb 2017 13:02:49 GMT

Hi,

We ran the HDFS balancer on one of the node in our environment but it fails to run on single node rather it started on entire cluster. 😞

We ran the balancer on single node by adding "-source -f HOSTS.TXT" to the command

Thanks.

Re: Hadoop Balancer

mqureshi — Wed, 08 Feb 2017 13:06:28 GMT

balancer is not for single node. It is for balancing load on the cluster. For balancing load among different disks on the same node, a new disk balancer will be available in Hadoop version 3.0 as the link Artem shared, shows. There is not much you would be able to do here, except for the fact that "don't worry, Hadoop is smart enough to know that particular disk doesn't have any more space and it will find another disk" 🙂

Re: Hadoop Balancer

rajsyrus — Wed, 08 Feb 2017 17:52:58 GMT

@Vijay

Hdfs balancer run across all the datanodes in appropriate distributing of the blocks, and the hadoop have the intelligent to shuffle the blocks in organized manner. so the balancer is not limited to sing node, you can just include and exclude the node for balancing. if you initiate the balancer in one node by excluding all but one, then its just like organizing the block on same node , so your disk utility will never reduce.

In your case just run the balancer accross the cluster, it will reduce the disk utility on the node, which is 99% filled.

Re: Hadoop Balancer

aervits — Wed, 08 Feb 2017 20:34:31 GMT

You also use -include -f with more than one host, not just that single datanode, I thought I was clear on that.

Re: Hadoop Balancer

reddyr211 — Thu, 09 Feb 2017 00:29:10 GMT

Thank you guys @mqureshi @Rajendra M @Artem Ervits

Re: Hadoop Balancer

aervits — Thu, 09 Feb 2017 00:53:05 GMT

absolutely, happy to help.

Re: Hadoop Balancer

rajsyrus — Mon, 13 Feb 2017 15:02:45 GMT

welcome,Thank you.

Re: Hadoop Balancer

SreeBalaje — Wed, 06 Nov 2019 18:09:26 GMT

In order to run a balancer on only one data node

hdfs balancer -include -f <#datanode name can be specified>

this would balance the data load on that particular DN.