Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hadoop Balancer

Solved Go to solution
Highlighted

Hadoop Balancer

Explorer

Hi ,

How to run Hadoop Balancer on a single node, in our present environment one of the data node is not balanced (disk usage at 99%) remaining all data nodes are at 55%. We are using HDP version(2.3.4)

We want to balance only that single node which is at 99%.

Thanks in Advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Hadoop Balancer

Mentor

are you asking about balancing HDFS so that all nodes are evenly distributed? In that case you need to run HDFS balancer and it will spread that node's data across nodes, otherwise it defeats the point of balancing out HDFS. You can use -include -f hostsfile to hint which nodes to run balancer against but idea is you need more than one node there, is meant to be across multiple datanodes not single datanode.

https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#balancer

If your question is to balance data across all disks on one node, you can use disk balancer, sadly it's a new feature in Hadoop 3. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html

View solution in original post

9 REPLIES 9
Highlighted

Re: Hadoop Balancer

Mentor

are you asking about balancing HDFS so that all nodes are evenly distributed? In that case you need to run HDFS balancer and it will spread that node's data across nodes, otherwise it defeats the point of balancing out HDFS. You can use -include -f hostsfile to hint which nodes to run balancer against but idea is you need more than one node there, is meant to be across multiple datanodes not single datanode.

https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#balancer

If your question is to balance data across all disks on one node, you can use disk balancer, sadly it's a new feature in Hadoop 3. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html

View solution in original post

Re: Hadoop Balancer

Explorer

Hi,

We ran the HDFS balancer on one of the node in our environment but it fails to run on single node rather it started on entire cluster. :(

We ran the balancer on single node by adding "-source -f HOSTS.TXT" to the command

Thanks.

Highlighted

Re: Hadoop Balancer

Super Guru

balancer is not for single node. It is for balancing load on the cluster. For balancing load among different disks on the same node, a new disk balancer will be available in Hadoop version 3.0 as the link Artem shared, shows. There is not much you would be able to do here, except for the fact that "don't worry, Hadoop is smart enough to know that particular disk doesn't have any more space and it will find another disk" :)

Highlighted

Re: Hadoop Balancer

Expert Contributor

@Vijay

Hdfs balancer run across all the datanodes in appropriate distributing of the blocks, and the hadoop have the intelligent to shuffle the blocks in organized manner. so the balancer is not limited to sing node, you can just include and exclude the node for balancing. if you initiate the balancer in one node by excluding all but one, then its just like organizing the block on same node , so your disk utility will never reduce.

In your case just run the balancer accross the cluster, it will reduce the disk utility on the node, which is 99% filled.

Highlighted

Re: Hadoop Balancer

Mentor

You also use -include -f with more than one host, not just that single datanode, I thought I was clear on that.

Highlighted

Re: Hadoop Balancer

Explorer
Highlighted

Re: Hadoop Balancer

Mentor

absolutely, happy to help.

Highlighted

Re: Hadoop Balancer

Expert Contributor

welcome,Thank you.

Highlighted

Re: Hadoop Balancer

New Contributor

In order to run a balancer on only one data node

hdfs balancer -include -f <#datanode name can be specified>

this would balance the data load on that particular DN.

Don't have an account?
Coming from Hortonworks? Activate your account here