question Re: What happens exactly when decommisioning a workner node? in Support Questions

What happens exactly when decommisioning a workner node?

Ludovic_Janssen — Wed, 21 Sep 2016 16:45:36 GMT

During the admin class two interesting questions on decommissioning worker nodes emerged:

- How does the decommissioning process stops the ApplicationMaster? Will it let the containers complete its current tasks or will it stop the container and recover the task on a different worker node? I assumed the last, but I would like to be sure.

- Will a decommissioned node still hold its physical data when being recommissioned? When is it removed? Does the removal occur at the first rebalancing when you initially decommission the node or will the data blocks still physically exist when you recommission the worker node?

Many thanks to clarify this!

Re: What happens exactly when decommisioning a workner node?

sbhat — Wed, 21 Sep 2016 23:04:18 GMT

@Ludovic Janssens

Please refer to the following doc :

https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Ambari_Users_Guide/content/_decommissioning_masters_and_slaves_.html

and

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_Sys_Admin_Guides/content/ref-b50b4ee6-0d7b-4b86-a06f-8e7bac00810f.1.html

to understand #1

To answer #2, Yes the physical data will remain in the worker node (unless you delete the node). You will need to rebalance once you recommission your node.

Refer point #7 here :

https://acadgild.com/blog/commissioning-and-decommissioning-of-datanode-in-hadoop/

Hope this helps!

Re: What happens exactly when decommisioning a workner node?

sedatkestepe — Thu, 14 Jun 2018 16:58:41 GMT

Does physical data really remain in the node?

For my case I saw lots of (thousands and repeating) HDFS log lines (on the node) about deleting blocks.

Are these lines unexpected (already have broken RAID controller)?

They keep being scheduled and deleted.

2018-06-14 11:58:53,005 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-1789482724-9.1.10.22-1491814552298 blk_1155905937_82210162 file /grid/2/hadoop/hdfs/data/current/BP-1789482724-9.1.10.22-1491814552298/current/finalized/subdir229/subdir185/blk_1155905937