Support Questions

Find answers, ask questions, and share your expertise

HDFS Balancer slow in CDH5.3.2

avatar
Contributor

Hi All,

 

We recently added 11 new datanodes to CDH5.3.2 cluster which has 20 datanodes.

I see HDFS Balancer slowly moving blocks say avg 4000 blocks peer 10 mins. I' updated Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

with 

 

<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>3048576000</value>
</property>

 

still i see alomst same speed, i checked network graphs also

 

Is there any other setting required?

 

Thanks,

Venu

1 ACCEPTED SOLUTION

avatar
Contributor

I opned case with cloudera and found solution

 

Add following settings in Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

 

<property>
<name>dfs.datanode.balance.max.concurrent.moves</name>
<value>300</value>
</property>

 

Thanks,

Venu

 

View solution in original post

3 REPLIES 3

avatar
Contributor

I opned case with cloudera and found solution

 

Add following settings in Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

 

<property>
<name>dfs.datanode.balance.max.concurrent.moves</name>
<value>300</value>
</property>

 

Thanks,

Venu

 

avatar
Contributor

I'm pretty sure that this configuration should go not to Balancer "Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", but in all Datanodes "Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" and DNs should be restarted after configuration changes.

avatar
Mentor
It should go to both of those valves with the same value currently, until HDFS-10289 gets done in future.