What happens exactly when decommisioning a workner node?

[#1]

During the admin class two interesting questions on decommissioning worker nodes emerged:

- How does the decommissioning process stops the ApplicationMaster? Will it let the containers complete its current tasks or will it stop the container and recover the task on a different worker node? I assumed the last, but I would like to be sure.

- Will a decommissioned node still hold its physical data when being recommissioned? When is it removed? Does the removal occur at the first rebalancing when you initially decommission the node or will the data blocks still physically exist when you recommission the worker node?

Many thanks to clarify this!

[#2 · Accepted solution]

@Ludovic Janssens

Please refer to the following doc :

https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Ambari_Users_Guide/content/_decommission...

and

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_Sys_Admin_Guides/content/ref-b50b4ee6-0d...

to understand #1

To answer #2, Yes the physical data will remain in the worker node (unless you delete the node). You will need to rebalance once you recommission your node.

Refer point #7 here :

https://acadgild.com/blog/commissioning-and-decommissioning-of-datanode-in-hadoop/

Hope this helps!

[#3]

Does physical data really remain in the node?

For my case I saw lots of (thousands and repeating) HDFS log lines (on the node) about deleting blocks.

Are these lines unexpected (already have broken RAID controller)?

They keep being scheduled and deleted.

2018-06-14 11:58:53,005 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(308)) - Deleted BP-1789482724-9.1.10.22-1491814552298 blk_1155905937_82210162 file /grid/2/hadoop/hdfs/data/current/BP-1789482724-9.1.10.22-1491814552298/current/finalized/subdir229/subdir185/blk_1155905937