HBase and HDFS Balancer

[#1]

Periodically, and when there are no regions in transition, a load balancer will run and move regions around to balance the cluster’s load. See Balancer for configuring this property. The Load Balancer ensures that the region replicas are not co-hosted in the same region servers and also in the same rack (if possible).

The HDFS balancer attempts to spread HDFS blocks evenly among DataNodes. HBase relies on compactions to restore locality after a region split or failure. These two types of balancing do not work well together.

In the past, the generally accepted advice was to turn off the HDFS load balancer and rely on the HBase balancer, since the HDFS balancer would degrade locality.

HDFS-6133 provides the ability to exclude favored-nodes (pinned) blocks from the HDFS load balancer, by setting thedfs.datanode.block-pinning.enabled property to true in the HDFS service configuration.

HBase can be enabled to use the HDFS favored-nodes feature by switching the HBase balancer class (conf: hbase.master.loadbalancer.class) to org.apache.hadoop.hbase.favored.FavoredNodeLoadBalancer which is documented here.

HDFS-6133 is available in HDFS 2.7.0 and higher, but HBase does not support running on HDFS 2.7.0, so you must be using HDFS 2.7.1 or higher to use this feature with HBase.

Reference: https://hbase.apache.org/book.html

[#2]

@Rohan Pednekar Could you provide pointers for HBASE APIs which should be used during data writing so that the data reaches only desired nodes ?

[#3]

@Rohan Pednekar

Issue

While I apply this article, I've encountered an exception on HBase configuration:

hbase.master.loadbalancer.class=org.apache.hadoop.hbase.favored.FavoredNodeLoadBalancer

Above configuration produces following exception:

java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.favored.FavoredNodeLoadBalancer not found

Solution

To fix this issue, reconfigure hbase.master.loadbalancer.class by changing the class name as followings:

hbase.master.loadbalancer.class=org.apache.hadoop.hbase.master.balancer.FavoredNodeLoadBalancer

Reference

Uses of Interface org.apache.hadoop.hbase.master.LoadBalancer