Support Questions
Find answers, ask questions, and share your expertise

HDFS Balancer slow in CDH5.3.2

Solved Go to solution

HDFS Balancer slow in CDH5.3.2

Explorer

Hi All,

 

We recently added 11 new datanodes to CDH5.3.2 cluster which has 20 datanodes.

I see HDFS Balancer slowly moving blocks say avg 4000 blocks peer 10 mins. I' updated Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

with 

 

<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>3048576000</value>
</property>

 

still i see alomst same speed, i checked network graphs also

 

Is there any other setting required?

 

Thanks,

Venu

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: HDFS Balancer slow in CDH5.3.2

Explorer

I opned case with cloudera and found solution

 

Add following settings in Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

 

<property>
<name>dfs.datanode.balance.max.concurrent.moves</name>
<value>300</value>
</property>

 

Thanks,

Venu

 

View solution in original post

3 REPLIES 3
Highlighted

Re: HDFS Balancer slow in CDH5.3.2

Explorer

I opned case with cloudera and found solution

 

Add following settings in Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

 

<property>
<name>dfs.datanode.balance.max.concurrent.moves</name>
<value>300</value>
</property>

 

Thanks,

Venu

 

View solution in original post

Highlighted

Re: HDFS Balancer slow in CDH5.3.2

Explorer

I'm pretty sure that this configuration should go not to Balancer "Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml", but in all Datanodes "Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" and DNs should be restarted after configuration changes.

Highlighted

Re: HDFS Balancer slow in CDH5.3.2

Master Guru
It should go to both of those valves with the same value currently, until HDFS-10289 gets done in future.
Don't have an account?