Created on 09-29-2017 01:45 PM - edited 09-16-2022 05:19 AM
Experiencing difficulty running the balancer after adding data nodes. The following message appears on most move attempts with a very small number of blocks succeding.
17/09/28 02:48:30 WARN balancer.Dispatcher: Failed to move blk_1073753574_12765 with size=134217728 from xxx.xxx.xxx.xxx:50010:DISK to xxx.xxx.xxx.xxx:50010:DISK through xxx.xxx.xxx.xxx Got error, status message opReplaceBlock BP-1162274874-xxx.xxx.xxx.xxx-1506099483351:blk_1073753574_12765 received exception java.io.IOException: Got error, status message Not able to copy block 1073753574 to /xxx.xxx.xxx.xxx:59796 because threads quota is exceeded., copy block BP-1162274874-xxx.xxx.xxx.xxx-1506099483351:blk_1073753574_12765 from /10.142.0.12:50010, block move is failed
Created 09-29-2017 01:45 PM
running hdp v2.6.2
Created 09-30-2017 01:22 AM
May I know what version of HDP are you using?
HDFS-6621 and officially released as part of Apache Hadoop 2.6.0. Since this is a bug in the Balancer itself, it is possible to run an updated version of the Balancer without upgrading your cluster. Datanodes will limit the number of threads used for balancing so as to not eat up all the resources of the cluster/datanode. This is what causes the WARN statement you're seeing. By default the number of threads is 5. This was not configurable prior to Apache Hadoop 2.5.0. HDFS-6595added this proeprty |
Hope this article helps in resolving balancer issue by running from commandline
https://community.hortonworks.com/questions/19694/help-with-exception-from-hdfs-balancer.html
Hope this helps you.