Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Node Decommissioning progressing too slowly

avatar
Rising Star

I'm decommissioning some storage heavy nodes that are taking a really long time (days) to move all the blocks over. There doesn't seem to be much out there showing how to increase the speed (http://stackoverflow.com/questions/17789196/hadoop-node-taking-a-long-time-to-decommission) but there must be something. At this rate it will take a weeks to decommission the required nodes

1 ACCEPTED SOLUTION

avatar
New Contributor

Use the following settings in custom hdfs-site.xml configuration from ambari for speeding up under-replicated blocks while decommissioning nodes:

dfs.namenode.replication.max-streams

dfs.namenode.replication.max-streams-hard-limit

dfs.namenode.replication.work.multiplier.per.iteration

defaults were 2,2,4 and I’ve set them to 50, 100, 200 just to test out the speed of replication and its shifting blocks quite quickly with these settings. I've observed spikes in data node's resource utilization. Use the settings depending on your cluster size, data node capacity and network bandwidth as this can cause huge load and can impact running jobs.

View solution in original post

2 REPLIES 2

avatar
New Contributor

Use the following settings in custom hdfs-site.xml configuration from ambari for speeding up under-replicated blocks while decommissioning nodes:

dfs.namenode.replication.max-streams

dfs.namenode.replication.max-streams-hard-limit

dfs.namenode.replication.work.multiplier.per.iteration

defaults were 2,2,4 and I’ve set them to 50, 100, 200 just to test out the speed of replication and its shifting blocks quite quickly with these settings. I've observed spikes in data node's resource utilization. Use the settings depending on your cluster size, data node capacity and network bandwidth as this can cause huge load and can impact running jobs.

avatar
Rising Star

Thanks! Will try. Its still in the early stages so load is not a huge concern right now