Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

Guru

Hello frndz,

Any idea why hdfs balancer is getting failed after 30 mins with following error.

resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"' ; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10'' returned 252. 16/03/08 08:42:03 INFO balancer.Balancer: Using a threshold of 10.0
16/03/08 08:42:03 INFO balancer.Balancer: namenodes  = [hdfs://HDPDEVHA]
16/03/08 08:42:03 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
16/03/08 08:42:03 INFO balancer.Balancer: included nodes = []
16/03/08 08:42:03 INFO balancer.Balancer: excluded nodes = []
16/03/08 08:42:03 INFO balancer.Balancer: source nodes = []
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
16/03/08 08:42:04 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
16/03/08 08:42:04 INFO block.BlockTokenSecretManager: Setting block keys
16/03/08 08:42:04 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
java.io.IOException: Another Balancer is running..  Exiting ...
Mar 8, 2016 8:42:04 AM   Balancing took 1.27 seconds
Last login: Tue Mar  8 08:12:09 EST 2016

And it balancing data on nodes,still I can see many data nodes are just having 0.04 utilised.

16 REPLIES 16

Re: hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

Mentor

What version of HDP is it, jdk version?

Highlighted

Re: hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

Guru

We are using jdk 1.7.

[root@samplea~]$ java -version

java version "1.7.0_67"

Java(TM) SE Runtime Environment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

Re: hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

Guru

I have hdp 2.3.

Re: hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

Mentor

@Saurabh Kumar

Do you have HA enabled for HDFS? If so, can you check the following property dfs.namenode.rpc-address, then

you are hitting a known issue but looks like your Ambari version should have the fix.

https://issues.apache.org/jira/browse/AMBARI-13373

WORKAROUND:

Use the configs.sh utility on the Ambari Server to delete the extra property.

Important: Only perform this procedure if your cluster is running NameNode HA and you are experiencing the HDFS balancer issue.

/var/lib/ambari-server/resources/scripts/configs.sh -u

<admin.user> -p

<admin.password> delete

<ambari.server>

<cluster.name> hdfs-site “dfs.namenode.rpc-address”

Where

admin.user and

admin.password are credentials for an Ambari Administrator,

ambari.server is the Ambari Server host and

cluster.name is the name of your cluster.

Re: hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

Mentor

Re: hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

Mentor

@Saurabh Kumar here's the document I was looking for, please make sure you delete any properties listed there for Ambari 2.2.0.0 https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Ambari_Users_Guide/content/_modify_hdfs_...

Re: hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

Guru

@Artem Ervits : I read given doc and understand it. But before I delete it from my HA enabled cluster, I just have one question.

If I will delete it then what would happened to client request,because afaik it use for RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.rpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port.

Can you please help me to understand it.

Re: hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

Mentor

You only delete the old properties not the ones with nn1 and nn2

Re: hdfs balancer is getting failed after 30 mins in ambari 2.2.0.

Guru

@Artem Ervits: I don't see old property "dfs.namenode.rpc-address" in hdfs-site.xml and was that issue in old ambari version or new version also. I am using ambari 2.2.0.

I have tried once again via command line also but unfortunately failed again with same error.

#excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]

16/03/08 08:42:03 INFO balancer.Balancer: included nodes = [] 16/03/08 08:42:03 INFO balancer.Balancer: excluded nodes = [] 16/03/08 08:42:03 INFO balancer.Balancer: source nodes = [] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved 16/03/08 08:42:04 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 16/03/08 08:42:04 INFO block.BlockTokenSecretManager: Setting block keys 16/03/08 08:42:04 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec java.io.IOException: Another Balancer is running.. Exiting ... Mar 8, 2016 8:42:04 AM Balancing took 1.27 seconds

Don't have an account?
Coming from Hortonworks? Activate your account here