02-09-2018 11:09 AM
Hi , I am planning to decommission a data node , is there any formula that can be used to estimate time it takes to decomission a node based on size per say the data that is there on my decommission node is around 1TB or datanode balancing Bandwidth ,Replication Work Multiplier Per Iteration ?.
02-10-2018 09:06 AM
I don't think any formula aviable to estimate the durtaion ( at least to my knowledge)
But I tried the data node decommission few times for various reasons and this is my experience
for a 40 TB node, it took between 45 mins and 60 mins approx (i hope it should be less than 5 or 10 mins for you)
you can monitor this from CM -> HDFS -> WebUI -> Namenode Web UI -> a new window will open, 'Datanodes' menu -> scroll down to Decommissioning (keep refresh this page to get the progress)
02-12-2018 08:24 AM
Thank you srinivas ,
is there any reason that you can think of why decommission takes longer ?, my decommission of host since past friday 3:00PM EST total data to be rebalanced is around 13TB,
02-12-2018 08:07 PM
srinivas ?? :-)
Is it struck when 1 or 2 blocks left over?
As mentioned earlier, you can monitor this from CM -> HDFS -> WebUI -> Namenode Web UI -> a new window will open, 'Datanodes' menu -> scroll down to Decommissioning (keep refresh this page to get the progress)
If your answer is yes for my above question, then I got the similar issues few times and I've over come this issue as follows:
1. CM -> Hosts -> Abort the decomm process
2. CM -> HDFS -> Instance -> Node -> Stop
3. Try to decommission the same node again for the left over blocks
Note: Some times you may struck again, retry couple of times
02-13-2018 06:53 PM
- Did you had a chance to raise the datanode bandwidh , Datanode heapsize , increase the replication work multiplier before kicking of the decommision . this will certainly increase the performance.
Also if your decommision is runining for ever i would suggest you to commission it back and perform decommision it again.