Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hdfs Balancer exits without balancing with error

Highlighted

Hdfs Balancer exits without balancing with error

New Contributor

Hello All,

HDP 2.6.3 Datanodes 3

I'm trying to balance my datanodes by running the below balancer command via CLI, I found that my balancer did not point to any of my Namenodes and also I found the below error.

devcluster is my Cluster name , masternode1dev is my Standby NN.

hdfs balancer -threshold 10

INFO balancer.Balancer: namenodes = [hdfs://devcluster]

INFO retry.RetryInvocationHandler: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby

at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)

at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2006)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1404)

at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getBlocks(BlockManager.java:1118)

at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlocks(NameNodeRpcServer.java:596)

at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.getBlocks(NamenodeProtocolServerSideTranslatorPB.java:88)

at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12828)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)

, while invoking NamenodeProtocolTranslatorPB.getBlocks over masternode1dev/xxx.xx.xx.xx:8020. Trying to failover immediately.

2 REPLIES 2

Re: Hdfs Balancer exits without balancing with error

@Joshva Peter

Above message "Operation category READ is not supported in state standby" is when the Balancer connects to Standby Namenode and the connection would later failover to other Active Namenode.

Since, there is no error reported. Capture the debug logs as below to analyse the issue.

export HADOOP_BALANCER_OPTS="-Droot.logger=DEBUG,console"

Run the balancer as:

hdfs balancer 'options' 2>&1 | tee /tmp/balancer.log

Re: Hdfs Balancer exits without balancing with error

New Contributor

@Sindhu

Output of hdfs balancer 'options'2>&1| tee /tmp/balancer.log

[hdfs@masternode2dev ~]$ cat /tmp/balancer.log Usage: hdfs balancer [-policy <policy>] the balancing policy: datanode or blockpool [-threshold <threshold>] Percentage of disk capacity [-exclude [-f <hosts-file> | <comma-separated list of hosts>]] Excludes the specified datanodes. [-include [-f <hosts-file> | <comma-separated list of hosts>]] Includes only the specified datanodes. [-source [-f <hosts-file> | <comma-separated list of hosts>]] Pick only the specified datanodes as source nodes. [-idleiterations <idleiterations>] Number of consecutive idle iterations (-1 for Infinite) before exit. [-runDuringUpgrade] Whether to run the balancer during an ongoing HDFS upgrade.This is usually not desired since it will not affect used space on over-utilized machines.

Jan 21, 2019 9:29:33 PM Balancing took 316.0 milliseconds

19/01/21 21:29:33 ERROR balancer.Balancer: Exiting balancer due an exception java.lang.IllegalArgumentException: args = [options]

at org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.parse(Balancer.java:876)

at org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:799)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)

at org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:929)

Don't have an account?
Coming from Hortonworks? Activate your account here