Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Node Decommissioning progressing too slowly

Solved Go to solution
Highlighted

Node Decommissioning progressing too slowly

Contributor

I'm decommissioning some storage heavy nodes that are taking a really long time (days) to move all the blocks over. There doesn't seem to be much out there showing how to increase the speed (http://stackoverflow.com/questions/17789196/hadoop-node-taking-a-long-time-to-decommission) but there must be something. At this rate it will take a weeks to decommission the required nodes

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Node Decommissioning progressing too slowly

New Contributor

Use the following settings in custom hdfs-site.xml configuration from ambari for speeding up under-replicated blocks while decommissioning nodes:

dfs.namenode.replication.max-streams

dfs.namenode.replication.max-streams-hard-limit

dfs.namenode.replication.work.multiplier.per.iteration

defaults were 2,2,4 and I’ve set them to 50, 100, 200 just to test out the speed of replication and its shifting blocks quite quickly with these settings. I've observed spikes in data node's resource utilization. Use the settings depending on your cluster size, data node capacity and network bandwidth as this can cause huge load and can impact running jobs.

2 REPLIES 2

Re: Node Decommissioning progressing too slowly

New Contributor

Use the following settings in custom hdfs-site.xml configuration from ambari for speeding up under-replicated blocks while decommissioning nodes:

dfs.namenode.replication.max-streams

dfs.namenode.replication.max-streams-hard-limit

dfs.namenode.replication.work.multiplier.per.iteration

defaults were 2,2,4 and I’ve set them to 50, 100, 200 just to test out the speed of replication and its shifting blocks quite quickly with these settings. I've observed spikes in data node's resource utilization. Use the settings depending on your cluster size, data node capacity and network bandwidth as this can cause huge load and can impact running jobs.

Re: Node Decommissioning progressing too slowly

Contributor

Thanks! Will try. Its still in the early stages so load is not a huge concern right now

Don't have an account?
Coming from Hortonworks? Activate your account here