Balancer not working in hdfs HA

[#1]

I was working on one of the case and came across an issue where balancer utility was not working in namenode HA environment.

HDP Version - 2.3.2.0

Ambari Version - 2.1.2

Issue - balancer was failing with below error:

[hdfs@slave1 ~]$ hdfs balancer 15/11/20 11:46:10 INFO balancer.Balancer: namenodes = [hdfs://mycluster, hdfs://mycluster:8020] 15/11/20 11:46:10 INFO balancer.Balancer: parameters = Balancer.Parameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, run during upgrade = false] [..] 15/11/20 11:46:11 INFO block.BlockTokenSecretManager: Setting block keys 15/11/20 11:46:11 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec java.io.IOException: Another Balancer is running.. Exiting ... Nov 20, 2015 11:46:11 AM Balancing took 1.883 seconds 

Resolution(workaround) -

1. Copy /etc/hadoop/conf/* /home/hdfs/copied_conf/

2. Edit /home/hdfs/copied_conf/hdfs-site.xml and delete below property

<property> 
 <name>dfs.namenode.rpc-address</name> 
 <value>xyz:8020</value>
</property> 

3. Run balancer with below command(as user hdfs):

hdfs --config /home/hdfs/copied_conf balancer 

Permanent Solution -

This will get fixed in Ambari 2.1.3. version (Refer - https://hortonworks.jira.com/browse/BUG-45749)

[#2]

@Shihab

That worked for me. Thanks so much. I also had to delete /system/diskbalancer.id to run it successfully. But for some reason I have to do this for every rebalancer I run.