hdfs balancer

[#1]

Experiencing difficulty running the balancer after adding data nodes. The following message appears on most move attempts with a very small number of blocks succeding.

17/09/28 02:48:30 WARN balancer.Dispatcher: Failed to move blk_1073753574_12765 with size=134217728 from xxx.xxx.xxx.xxx:50010:DISK to xxx.xxx.xxx.xxx:50010:DISK through xxx.xxx.xxx.xxx Got error, status message opReplaceBlock BP-1162274874-xxx.xxx.xxx.xxx-1506099483351:blk_1073753574_12765 received exception java.io.IOException: Got error, status message Not able to copy block 1073753574 to /xxx.xxx.xxx.xxx:59796 because threads quota is exceeded., copy block BP-1162274874-xxx.xxx.xxx.xxx-1506099483351:blk_1073753574_12765 from /10.142.0.12:50010, block move is failed

[#2]

running hdp v2.6.2

[#3]

@frank policano

May I know what version of HDP are you using?

HDFS-6621 and officially released as part of Apache Hadoop 2.6.0. Since this is a bug in the Balancer itself, it is possible to run an updated version of the Balancer without upgrading your cluster.

Datanodes will limit the number of threads used for balancing so as to not eat up all the resources of the cluster/datanode. This is what causes the WARN statement you're seeing. By default the number of threads is 5. This was not configurable prior to Apache Hadoop 2.5.0. HDFS-6595added this proeprty dfs.datanode.balance.max.concurrent.moves to allow you to control the number of threads used for balancing. Since this is a datanode side property, this will require an upgrade to your cluster if you want to use this setting.

https://stackoverflow.com/questions/25222633/hadoop-balancer-command-warn-messages-threads-quota-is-...

Hope this article helps in resolving balancer issue by running from commandline

https://community.hortonworks.com/questions/19694/help-with-exception-from-hdfs-balancer.html

Hope this helps you.