Reply
Explorer
Posts: 11
Registered: ‎04-13-2015
Accepted Solution

HDFS Balancer slow in CDH5.3.2

Hi All,

 

We recently added 11 new datanodes to CDH5.3.2 cluster which has 20 datanodes.

I see HDFS Balancer slowly moving blocks say avg 4000 blocks peer 10 mins. I' updated Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

with 

 

<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>3048576000</value>
</property>

 

still i see alomst same speed, i checked network graphs also

 

Is there any other setting required?

 

Thanks,

Venu

Explorer
Posts: 11
Registered: ‎04-13-2015

Re: HDFS Balancer slow in CDH5.3.2

I opned case with cloudera and found solution

 

Add following settings in Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

 

<property>
<name>dfs.datanode.balance.max.concurrent.moves</name>
<value>300</value>
</property>

 

Thanks,

Venu

 

Highlighted
Explorer
Posts: 13
Registered: ‎11-21-2013

Re: HDFS Balancer slow in CDH5.3.2

[ Edited ]

I'm pretty sure that this configuration should go not to Balancer "Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", but in all Datanodes "Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" and DNs should be restarted after configuration changes.

Posts: 1,880
Kudos: 422
Solutions: 297
Registered: ‎07-31-2013

Re: HDFS Balancer slow in CDH5.3.2

It should go to both of those valves with the same value currently, until HDFS-10289 gets done in future.
Announcements