time for decommission a data node

[#1]

Hi , I am planning to decommission a data node , is there any formula that can be used to estimate time it takes to decomission a node based on size per say the data that is there on my decommission node is around 1TB or datanode balancing Bandwidth ,Replication Work Multiplier Per Iteration ?.

Thanks

[#2]

@Cloudera learning

I don't think any formula aviable to estimate the durtaion ( at least to my knowledge)

But I tried the data node decommission few times for various reasons and this is my experience

for a 40 TB node, it took between 45 mins and 60 mins approx (i hope it should be less than 5 or 10 mins for you)

you can monitor this from CM -> HDFS -> WebUI -> Namenode Web UI -> a new window will open, 'Datanodes' menu -> scroll down to Decommissioning (keep refresh this page to get the progress)

[#3]

Thank you srinivas ,

is there any reason that you can think of why decommission takes longer ?, my decommission of host since past friday 3:00PM EST total data to be rebalanced is around 13TB,

[#4]

srinivas ?? 🙂

@Cloudera learning

Is it struck when 1 or 2 blocks left over? 

As mentioned earlier, you can monitor this from CM -> HDFS -> WebUI -> Namenode Web UI -> a new window will open, 'Datanodes' menu -> scroll down to Decommissioning (keep refresh this page to get the progress)

If your answer is yes for my above question, then I got the similar issues few times and I've over come this issue as follows:

1. CM -> Hosts -> Abort the decomm process

2. CM -> HDFS -> Instance -> Node -> Stop

3. Try to decommission the same node again for the left over blocks

Note: Some times you may struck again, retry couple of times

[#5]

@Cloudera learning

- Did you had a chance to raise the datanode bandwidh , Datanode heapsize , increase the replication work multiplier before kicking of the decommision . this will certainly increase the performance. 

Also if your decommision is runining for ever  i would suggest you to commission it back and perform decommision it again. 
-