Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

Rebalance HDFS - Not able to copy block 1314213952 because threads quota is exceeded


We are running rebalance on HDFS and we are getting the error below:


"WARN balancer.Dispatcher: Failed to move blk_1314197946_240487461 with size=134217728 from to through Got error, status message opReplaceBlock BP-1707938289- received exception Got error, status message Not able to copy block 1314197946 to / because threads quota is exceeded., copy block BP-1707938289- from /, block move is failed"


we would like to know why we are getting these errors?


would it be because the network interfaces are 1GB?


if so, how can we resolve these errors? increasing the speed of network interfaces?


PS: The hosts have 1GB network interfaces

PS: We are using Ambari Server version


Cloudera Employee


The error is due to the exhausted thread quota on the DN side. Usually this can be controlled using the balancer parameters. 

Kindly refer

Ideally changing the values for "dfs.datanode.balance.max.concurrent.moves" should help come out of the issue. 

n/w bandwidth can become an issue while we are dealing with large volume of data movement but according to this error it's on the quota.


hi @Akarsh 


the documentation doesn't show how to use the parameter "dfs.datanode.balance.max.concurrent.moves" in the balancer execution command.


how can we use this parameter in the balancer execution?

Cloudera Employee

There are 2 ways. One is directly adding it in hdfs-site.xml OR just triggering the balancer with these parameters like 

nohup hdfs balancer -Ddfs.balancer.moverThreads=300 -Ddfs.datanode.balance.max.concurrent.moves=20 -Ddfs.datanode.balance.bandwidthPerSec=20480000 -Ddfs.balancer.dispatcherThreads=400 -Ddfs.balancer.max-size-to-move=100737418240 -threshold 10 >/tmp/new_balancer1.out 
This will run the balancer in non default values and it will finish the balancer operation much more quicker. 
** Be aware that the run using above command and parameter will cause high Bandwidth usage and will create lot of i/o storms. 

For more details on the parameters mentioned above please refer below doc



hi @Akarsh 


The balancer was run like this:


hdfs balancer -Ddfs.datanode.balance.bandwidthPerSec=1073741824 -Ddfs.datanode.balance.max.concurrent.moves=20 -threshold 5


But still the same error appeared:


WARN balancer.Dispatcher: Failed to move blk_1275620781_201901979 with size=99352364 from to through Got error, status message Not able to receive block 1275620781 from / because threads quota is exceeded., block move is failed

Cloudera Employee

Hi @yagoaparecidoti 


You need not need to worry about this warning as these blocks will be re-attempted to move again during the balancer job run.


For example, check after several hours, regarding the block "blk_1275620781" movement completion in logs.


hi @shubham_sharma, how are you?


thanks for the info.


we will rerun the rebalancer on HDFS and monitor the behavior.


any error that appears we return in the sequence.

Community Manager

@yagoaparecidoti Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  Thanks!


Diana Torres,
Community Moderator

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community: