Reply
New Contributor
Posts: 2
Registered: ‎12-21-2015

time for decommission a data node

Hi , I am planning to decommission a data node , is there any formula that can be used to estimate time it takes to decomission a node based on size per say the data that is there on my decommission node is around 1TB or datanode balancing Bandwidth ,Replication Work Multiplier Per Iteration ?.

 

 

Thanks

 

 

 

Highlighted
Posts: 418
Topics: 13
Kudos: 67
Solutions: 39
Registered: ‎09-02-2016

Re: time for decommission a data node

@Cloudera learning

 

I don't think any formula aviable to estimate the durtaion ( at least to my knowledge)

 

But I tried the data node decommission few times for various reasons and this is my experience

 

for a 40 TB node, it took between 45 mins and 60 mins approx (i hope it should be less than 5 or 10 mins for you)

 

you can monitor this from CM -> HDFS -> WebUI -> Namenode Web UI -> a new window will open, 'Datanodes' menu -> scroll down to Decommissioning (keep refresh this page to get the progress)

New Contributor
Posts: 2
Registered: ‎12-21-2015

Re: time for decommission a data node

Thank you srinivas ,

 

is there any reason that you can think of why decommission takes longer ?, my decommission of host since past friday 3:00PM EST total data to be rebalanced is around 13TB,

Posts: 418
Topics: 13
Kudos: 67
Solutions: 39
Registered: ‎09-02-2016

Re: time for decommission a data node

 

srinivas ?? :-)

 

@Cloudera learning

 

Is it struck when 1 or 2 blocks left over? 

 

As mentioned earlier, you can monitor this from CM -> HDFS -> WebUI -> Namenode Web UI -> a new window will open, 'Datanodes' menu -> scroll down to Decommissioning (keep refresh this page to get the progress)

 

If your answer is yes for my above question, then I got the similar issues few times and I've over come this issue as follows:

 

1. CM -> Hosts -> Abort the decomm process

2. CM -> HDFS -> Instance -> Node -> Stop

3. Try to decommission the same node again for the left over blocks

 

Note: Some times you may struck again, retry couple of times

Champion
Posts: 642
Registered: ‎05-16-2016

Re: time for decommission a data node

@Cloudera learning

 

- Did you had a chance to raise the datanode bandwidh , Datanode heapsize , increase the replication work multiplier before kicking of the decommision . this will certainly increase the performance. 

Also if your decommision is runining for ever  i would suggest you to commission it back and perform decommision it again. 
-

Announcements