Member since
04-04-2016
1
Post
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
7779 | 05-12-2017 01:08 PM |
05-12-2017
01:08 PM
2 Kudos
Use the following settings in custom hdfs-site.xml configuration from ambari for speeding up under-replicated blocks while decommissioning nodes: dfs.namenode.replication.max-streams dfs.namenode.replication.max-streams-hard-limit dfs.namenode.replication.work.multiplier.per.iteration defaults were 2,2,4 and I’ve set them to 50, 100, 200 just to test out the speed of replication and its shifting blocks quite quickly with these settings. I've observed spikes in data node's resource utilization. Use the settings depending on your cluster size, data node capacity and network bandwidth as this can cause huge load and can impact running jobs.
... View more