Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hadoop Balancer

avatar
Contributor

Hi ,

How to run Hadoop Balancer on a single node, in our present environment one of the data node is not balanced (disk usage at 99%) remaining all data nodes are at 55%. We are using HDP version(2.3.4)

We want to balance only that single node which is at 99%.

Thanks in Advance.

1 ACCEPTED SOLUTION

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
9 REPLIES 9

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Contributor

Hi,

We ran the HDFS balancer on one of the node in our environment but it fails to run on single node rather it started on entire cluster. 😞

We ran the balancer on single node by adding "-source -f HOSTS.TXT" to the command

Thanks.

avatar
Super Guru

balancer is not for single node. It is for balancing load on the cluster. For balancing load among different disks on the same node, a new disk balancer will be available in Hadoop version 3.0 as the link Artem shared, shows. There is not much you would be able to do here, except for the fact that "don't worry, Hadoop is smart enough to know that particular disk doesn't have any more space and it will find another disk" 🙂

avatar
Expert Contributor

@Vijay

Hdfs balancer run across all the datanodes in appropriate distributing of the blocks, and the hadoop have the intelligent to shuffle the blocks in organized manner. so the balancer is not limited to sing node, you can just include and exclude the node for balancing. if you initiate the balancer in one node by excluding all but one, then its just like organizing the block on same node , so your disk utility will never reduce.

In your case just run the balancer accross the cluster, it will reduce the disk utility on the node, which is 99% filled.

avatar
Master Mentor

You also use -include -f with more than one host, not just that single datanode, I thought I was clear on that.

avatar
Contributor

avatar
Master Mentor

absolutely, happy to help.

avatar
Expert Contributor

welcome,Thank you.

avatar
New Contributor

In order to run a balancer on only one data node

hdfs balancer -include -f <#datanode name can be specified>

this would balance the data load on that particular DN.