Created on 11-23-2015 10:37 AM - edited 09-16-2022 01:33 AM
I was working on one of the case and came across an issue where balancer utility was not working in namenode HA environment.
HDP Version - 2.3.2.0
Ambari Version - 2.1.2
Issue - balancer was failing with below error:
[hdfs@slave1 ~]$ hdfs balancer 15/11/20 11:46:10 INFO balancer.Balancer: namenodes = [hdfs://mycluster, hdfs://mycluster:8020] 15/11/20 11:46:10 INFO balancer.Balancer: parameters = Balancer.Parameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, run during upgrade = false] [..] 15/11/20 11:46:11 INFO block.BlockTokenSecretManager: Setting block keys 15/11/20 11:46:11 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec java.io.IOException: Another Balancer is running.. Exiting ... Nov 20, 2015 11:46:11 AM Balancing took 1.883 seconds
Resolution(workaround) -
1. Copy /etc/hadoop/conf/* /home/hdfs/copied_conf/
2. Edit /home/hdfs/copied_conf/hdfs-site.xml and delete below property
<property> <name>dfs.namenode.rpc-address</name> <value>xyz:8020</value> </property>3. Run balancer with below command(as user hdfs):
hdfs --config /home/hdfs/copied_conf balancer
Permanent Solution -
This will get fixed in Ambari 2.1.3. version (Refer - https://hortonworks.jira.com/browse/BUG-45749)
Created on 11-23-2015 11:26 AM
thank you @Kuldeep Kulkarni I have same issue with a prospect. Same happens to hdfs mover.
Created on 11-23-2015 12:51 PM
@Guilherme Braccialli - Glad this helped! 🙂
Created on 11-23-2015 04:36 PM
I also found this jira: https://issues.apache.org/jira/browse/AMBARI-13946
Created on 11-23-2015 07:30 PM
@Kuldeep Kulkarni another workaround:
1- Execute commands below:
/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p AMBARI-PASSWORD delete localhost CLUSTER-NAME hdfs-site "dfs.namenode.rpc-address" /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p AMBARI-PASSWORD delete localhost CLUSTER-NAME hdfs-site "dfs.namenode.http-address" /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p AMBARI-PASSWORD delete localhost CLUSTER-NAME hdfs-site "dfs.namenode.https-address"
2- Restart HDFS
Created on 11-23-2015 08:05 PM
Thanks @Guilherme Braccialli
Created on 11-24-2015 04:09 AM
Great! I believe ambari and hdfs services restart is needed if we delete configs using configs.sh script.
Created on 11-24-2015 05:09 AM
@Kuldeep Kulkarni ambari restart is not needed, I tested and updated configs appears immediately. You are right about HDFS restart, also mentioned as step #2.
Created on 11-24-2015 06:05 AM
thanks @Guilherme Braccialli - that makes sense.
Created on 07-14-2016 12:20 PM
We ran into the same issue using Ambari 2.2.2.2.It stops at 35%.Says another instance is running and exiting. when you restart form Ambari or from the command line .Fix is :go to your name node run ps -aef|grep balancer kill -9 all the balancer process except the irq balancer (Linux native).Start the balancer from cmd line using # sudo -u hdfs hdfs balancer it works !!!
Created on 01-09-2017 07:12 PM
I see the error in HDP 2.4.2, Ambari 2.2.2.0 version as well. HA is enabled. But as per the fix, it should be resolved in this version of ambari right... ?? Do you know why this error could be persisting ??