Skip to main content
KuldeepK
Gold ⭐️
KuldeepK
Gold ⭐️
November 23, 2015

Balancer not working in hdfs HA

  • October 5, 2026 at 9:29 AM
  • 11 replies
  • 17562 views
  • Translate All

I was working on one of the case and came across an issue where balancer utility was not working in namenode HA environment.

HDP Version - 2.3.2.0

Ambari Version - 2.1.2

Issue - balancer was failing with below error:

[hdfs@slave1 ~]$ hdfs balancer 15/11/20 11:46:10 INFO balancer.Balancer: namenodes = [hdfs://mycluster, hdfs://mycluster:8020] 15/11/20 11:46:10 INFO balancer.Balancer: parameters = Balancer.Parameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, run during upgrade = false] [..] 15/11/20 11:46:11 INFO block.BlockTokenSecretManager: Setting block keys 15/11/20 11:46:11 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec java.io.IOException: Another Balancer is running.. Exiting ... Nov 20, 2015 11:46:11 AM Balancing took 1.883 seconds

Resolution(workaround) -

1. Copy /etc/hadoop/conf/* /home/hdfs/copied_conf/

2. Edit /home/hdfs/copied_conf/hdfs-site.xml and delete below property

<property>
<name>dfs.namenode.rpc-address</name>
<value>xyz:8020</value>
</property>

3. Run balancer with below command(as user hdfs):

hdfs --config /home/hdfs/copied_conf balancer

Permanent Solution -

This will get fixed in Ambari 2.1.3. version (Refer - https://hortonworks.jira.com/browse/BUG-45749)

    11 replies

    gbraccialli3
    gbraccialli3
    Gold ⭐️
    November 23, 2015

    thank you @Kuldeep Kulkarni I have same issue with a prospect. Same happens to hdfs mover.

    Translate
      KuldeepK
      Gold ⭐️
      KuldeepKAuthor
      Gold ⭐️
      November 23, 2015

      @Guilherme Braccialli - Glad this helped! 🙂

      Translate
      gbraccialli3
      gbraccialli3
      Gold ⭐️
      November 24, 2015

      I also found this jira: https://issues.apache.org/jira/browse/AMBARI-13946

      Translate
        gbraccialli3
        gbraccialli3
        Gold ⭐️
        November 24, 2015

        @Kuldeep Kulkarni another workaround:

        1- Execute commands below:

        /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p AMBARI-PASSWORD delete localhost CLUSTER-NAME hdfs-site "dfs.namenode.rpc-address"
        /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p AMBARI-PASSWORD delete localhost CLUSTER-NAME hdfs-site "dfs.namenode.http-address"
        /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p AMBARI-PASSWORD delete localhost CLUSTER-NAME hdfs-site "dfs.namenode.https-address"

        2- Restart HDFS

        Translate
          nsabharwal
          nsabharwal
          Gold ⭐️
          November 24, 2015

          Thanks @Guilherme Braccialli

          Translate
            shihab_pri
            shihab_pri
            Gold ⭐️
            July 14, 2016

            We ran into the same issue using Ambari 2.2.2.2.It stops at 35%.Says another instance is running and exiting. when you restart form Ambari or from the command line .Fix is :go to your name node run ps -aef|grep balancer kill -9 all the balancer process except the irq balancer (Linux native).Start the balancer from cmd line using # sudo -u hdfs hdfs balancer it works !!!

            Translate
            pmj
            pmj
            Gold ⭐️
            January 10, 2017

            @Kuldeep Kulkarni

            I see the error in HDP 2.4.2, Ambari 2.2.2.0 version as well. HA is enabled. But as per the fix, it should be resolved in this version of ambari right... ?? Do you know why this error could be persisting ??

            Translate
            pmj
            pmj
            Gold ⭐️
            January 11, 2017

            @Shihab

            That worked for me. Thanks so much. I also had to delete /system/diskbalancer.id to run it successfully. But for some reason I have to do this for every rebalancer I run.

            Translate
            Terms & ConditionsAccessibility statement