question Re: HDFS Balancer slow in CDH5.3.2 in Support Questions

HDFS Balancer slow in CDH5.3.2

venu123 — Fri, 16 Sep 2022 09:45:07 GMT

Hi All,

We recently added 11 new datanodes to CDH5.3.2 cluster which has 20 datanodes.

I see HDFS Balancer slowly moving blocks say avg 4000 blocks peer 10 mins. I' updated Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

with

<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>3048576000</value>
</property>

still i see alomst same speed, i checked network graphs also

Is there any other setting required?

Thanks,

Venu

Re: HDFS Balancer slow in CDH5.3.2

venu123 — Mon, 26 Oct 2015 04:21:34 GMT

I opned case with cloudera and found solution

Add following settings in Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

<property>
<name>dfs.datanode.balance.max.concurrent.moves</name>
<value>300</value>
</property>

Thanks,

Venu

Re: HDFS Balancer slow in CDH5.3.2

Minutis — Mon, 14 Nov 2016 11:31:46 GMT

I'm pretty sure that this configuration should go not to Balancer "Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", but in all Datanodes "Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" and DNs should be restarted after configuration changes.

Re: HDFS Balancer slow in CDH5.3.2

Harsh J — Mon, 14 Nov 2016 11:34:51 GMT

It should go to both of those valves with the same value currently, until HDFS-10289 gets done in future.