I had an issue with datanodes, resulting in having about 300k under replicated blocks.
DNs are back, blocks are being replicated, but this is very slow, about 1 per second, and I am trying to find a way to speed replication up.
I checked dfs.datanode.balance.bandwidthPerSec, which is set at about 6MB/seconds. My monitoring shows me that on average the rx/tx from each node is about 200k/seconds, so I am way below this limit.
I followed this link https://community.hortonworks.com/articles/4427/fix-under-replicated-blocks-in-hdfs-manually.html which did not help (use setrep -w 3 on all underreplicated files )
This link https://community.hortonworks.com/questions/57795/how-to-fix-under-replicated-blocks-fasly-its-take.... is not fully applicable (hadoop 2.7) but I set fs.namenode.replication.work.multiplier.per.iteration to 100 (default is 2) without visible speed up.
So my question is: what can I do to fasten replication up?
Context: hdp2.6, AWS, 3 nodes and replication factor = 3.
The answer pointed at https://community.hortonworks.com/questions/57795/how-to-fix-under-replicated-blocks-fasly-its-take.... is the good one. Those are undocumented features in hadoop 2.7 but they can be set up and used and now I do see that replication is speed up.
View solution in original post