question Re: Node Decommissioning progressing too slowly in Archives of Support Questions (Read Only)

Node Decommissioning progressing too slowly

scarroll — Fri, 12 May 2017 20:02:37 GMT

I'm decommissioning some storage heavy nodes that are taking a really long time (days) to move all the blocks over. There doesn't seem to be much out there showing how to increase the speed (http://stackoverflow.com/questions/17789196/hadoop-node-taking-a-long-time-to-decommission) but there must be something. At this rate it will take a weeks to decommission the required nodes

Re: Node Decommissioning progressing too slowly

bmaragalla — Fri, 12 May 2017 20:08:44 GMT

Use the following settings in custom hdfs-site.xml configuration from ambari for speeding up under-replicated blocks while decommissioning nodes:

dfs.namenode.replication.max-streams

dfs.namenode.replication.max-streams-hard-limit

dfs.namenode.replication.work.multiplier.per.iteration

defaults were 2,2,4 and I’ve set them to 50, 100, 200 just to test out the speed of replication and its shifting blocks quite quickly with these settings. I've observed spikes in data node's resource utilization. Use the settings depending on your cluster size, data node capacity and network bandwidth as this can cause huge load and can impact running jobs.

Re: Node Decommissioning progressing too slowly

scarroll — Fri, 12 May 2017 20:21:10 GMT

Thanks! Will try. Its still in the early stages so load is not a huge concern right now