Reply
New Contributor
Posts: 2
Registered: ‎12-21-2015

time for decommission a data node

Hi , I am planning to decommission a data node , is there any formula that can be used to estimate time it takes to decomission a node based on size per say the data that is there on my decommission node is around 1TB or datanode balancing Bandwidth ,Replication Work Multiplier Per Iteration ?.

 

 

Thanks

 

 

 

Posts: 454
Topics: 13
Kudos: 75
Solutions: 41
Registered: ‎09-02-2016

Re: time for decommission a data node

@Cloudera learning

 

I don't think any formula aviable to estimate the durtaion ( at least to my knowledge)

 

But I tried the data node decommission few times for various reasons and this is my experience

 

for a 40 TB node, it took between 45 mins and 60 mins approx (i hope it should be less than 5 or 10 mins for you)

 

you can monitor this from CM -> HDFS -> WebUI -> Namenode Web UI -> a new window will open, 'Datanodes' menu -> scroll down to Decommissioning (keep refresh this page to get the progress)

New Contributor
Posts: 2
Registered: ‎12-21-2015

Re: time for decommission a data node

Thank you srinivas ,

 

is there any reason that you can think of why decommission takes longer ?, my decommission of host since past friday 3:00PM EST total data to be rebalanced is around 13TB,

Highlighted
Posts: 454
Topics: 13
Kudos: 75
Solutions: 41
Registered: ‎09-02-2016

Re: time for decommission a data node

 

srinivas ?? :-)

 

@Cloudera learning

 

Is it struck when 1 or 2 blocks left over? 

 

As mentioned earlier, you can monitor this from CM -> HDFS -> WebUI -> Namenode Web UI -> a new window will open, 'Datanodes' menu -> scroll down to Decommissioning (keep refresh this page to get the progress)

 

If your answer is yes for my above question, then I got the similar issues few times and I've over come this issue as follows:

 

1. CM -> Hosts -> Abort the decomm process

2. CM -> HDFS -> Instance -> Node -> Stop

3. Try to decommission the same node again for the left over blocks

 

Note: Some times you may struck again, retry couple of times

Champion
Posts: 714
Registered: ‎05-16-2016

Re: time for decommission a data node

@Cloudera learning

 

- Did you had a chance to raise the datanode bandwidh , Datanode heapsize , increase the replication work multiplier before kicking of the decommision . this will certainly increase the performance. 

Also if your decommision is runining for ever  i would suggest you to commission it back and perform decommision it again. 
-

Announcements