Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDFS Balancer not running in NN HA MODE

HDFS Balancer not running in NN HA MODE

New Contributor

Team,

HDFS balancer is not running on HA enabled cluster . Please help

:~]! hdfs balancer -threshold 10

17/01/09 07:11:39 INFO balancer.Balancer: Using a threshold of 10.0

17/01/09 07:11:39 INFO balancer.Balancer: namenodes = [hdfs://***-cluster:8020, hdfs://**-cluster]

17/01/09 07:11:39 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]

17/01/09 07:11:39 INFO balancer.Balancer: included nodes = []

17/01/09 07:11:39 INFO balancer.Balancer: excluded nodes = []

17/01/09 07:11:39 INFO balancer.Balancer: source nodes = []

Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved

17/01/09 07:11:41 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec

17/01/09 07:11:41 INFO block.BlockTokenSecretManager: Setting block keys

17/01/09 07:11:41 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec

17/01/09 07:11:41 INFO block.BlockTokenSecretManager: Setting block keys

17/01/09 07:11:41 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec

17/01/09 07:11:41 INFO block.BlockTokenSecretManager: Setting block keys

17/01/09 07:11:41 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec

java.io.IOException: Another Balancer is running.. Exiting ... **********

Jan 9, 2017 7:11:41 AM Balancing took 2.363 seconds

3 REPLIES 3

Re: HDFS Balancer not running in NN HA MODE

Super Collaborator

@suresh krish

Error shows that another balancer is already running. Check if there is a balancer process on namenode. And check for hdfs files #hdfs dfs -ls /system/balancer.id

#hdfs dfs -cat /system/balancer.id

If this file exists then there is another balancer running.

Re: HDFS Balancer not running in NN HA MODE

New Contributor

No OTher process is running on the cluster.. I can see through ambari

Re: HDFS Balancer not running in NN HA MODE

Contributor

@suresh krish

This is due to Ambari Bug https://issues.apache.org/jira/browse/AMBARI-13946

We have to remove these properties manually via configs.sh script.

dfs.namenode.rpc-address 
dfs.namenode.http-address 
dfs.namenode.https-address 

Run this on your ambari server

/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin delete localhost ironman hdfs-site dfs.namenode.rpc-address 
/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin delete localhost ironman hdfs-site dfs.namenode.http-address 
/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin delete localhost ironman hdfs-site dfs.namenode.https-address 

Where

localhost is your Ambari Host

Ironman is your cluster name

After making these changes, Ambari will show restart on HDFS services. Restart the services and you should be good to go.