Fix under replicated blocks very slow

[#1]

Hi,

I had an issue with datanodes, resulting in having about 300k under replicated blocks.

DNs are back, blocks are being replicated, but this is very slow, about 1 per second, and I am trying to find a way to speed replication up.

I checked dfs.datanode.balance.bandwidthPerSec, which is set at about 6MB/seconds. My monitoring shows me that on average the rx/tx from each node is about 200k/seconds, so I am way below this limit.

I followed this link https://community.hortonworks.com/articles/4427/fix-under-replicated-blocks-in-hdfs-manually.html which did not help (use setrep -w 3 on all underreplicated files )

This link https://community.hortonworks.com/questions/57795/how-to-fix-under-replicated-blocks-fasly-its-take.... is not fully applicable (hadoop 2.7) but I set fs.namenode.replication.work.multiplier.per.iteration to 100 (default is 2) without visible speed up.

So my question is: what can I do to fasten replication up?

Context: hdp2.6, AWS, 3 nodes and replication factor = 3.

[#2 · Accepted solution]

The answer pointed at https://community.hortonworks.com/questions/57795/how-to-fix-under-replicated-blocks-fasly-its-take.... is the good one. Those are undocumented features in hadoop 2.7 but they can be set up and used and now I do see that replication is speed up.