Node Decommissioning progressing too slowly

[#1]

I'm decommissioning some storage heavy nodes that are taking a really long time (days) to move all the blocks over. There doesn't seem to be much out there showing how to increase the speed (http://stackoverflow.com/questions/17789196/hadoop-node-taking-a-long-time-to-decommission) but there must be something. At this rate it will take a weeks to decommission the required nodes

[#2 · Accepted solution]

Use the following settings in custom hdfs-site.xml configuration from ambari for speeding up under-replicated blocks while decommissioning nodes:

dfs.namenode.replication.max-streams

dfs.namenode.replication.max-streams-hard-limit

dfs.namenode.replication.work.multiplier.per.iteration

defaults were 2,2,4 and I’ve set them to 50, 100, 200 just to test out the speed of replication and its shifting blocks quite quickly with these settings. I've observed spikes in data node's resource utilization. Use the settings depending on your cluster size, data node capacity and network bandwidth as this can cause huge load and can impact running jobs.

[#3]

Thanks! Will try. Its still in the early stages so load is not a huge concern right now