Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Fix under replicated blocks very slow

Solved Go to solution
Highlighted

Fix under replicated blocks very slow

Expert Contributor

Hi,

I had an issue with datanodes, resulting in having about 300k under replicated blocks.

DNs are back, blocks are being replicated, but this is very slow, about 1 per second, and I am trying to find a way to speed replication up.

I checked dfs.datanode.balance.bandwidthPerSec, which is set at about 6MB/seconds. My monitoring shows me that on average the rx/tx from each node is about 200k/seconds, so I am way below this limit.

I followed this link https://community.hortonworks.com/articles/4427/fix-under-replicated-blocks-in-hdfs-manually.html which did not help (use setrep -w 3 on all underreplicated files )

This link https://community.hortonworks.com/questions/57795/how-to-fix-under-replicated-blocks-fasly-its-take.... is not fully applicable (hadoop 2.7) but I set fs.namenode.replication.work.multiplier.per.iteration to 100 (default is 2) without visible speed up.

So my question is: what can I do to fasten replication up?

Context: hdp2.6, AWS, 3 nodes and replication factor = 3.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Fix under replicated blocks very slow

Expert Contributor

The answer pointed at https://community.hortonworks.com/questions/57795/how-to-fix-under-replicated-blocks-fasly-its-take.... is the good one. Those are undocumented features in hadoop 2.7 but they can be set up and used and now I do see that replication is speed up.

View solution in original post

1 REPLY 1
Highlighted

Re: Fix under replicated blocks very slow

Expert Contributor

The answer pointed at https://community.hortonworks.com/questions/57795/how-to-fix-under-replicated-blocks-fasly-its-take.... is the good one. Those are undocumented features in hadoop 2.7 but they can be set up and used and now I do see that replication is speed up.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here