Replacing Disks on Datanode Hosts

[#1]

Below are steps to replace disk in slave nodes or to perform maintenance of slave nodes servers.

1. Decommission the Datanode and all services running on it (i.e. NodeManager, HBase RegionServer, Datanode etc). Refer below docs for the same.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-user-guide/content/decommissionin...

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_administration/content/ch_slave_nodes.ht...

2. Replace the disks or perform any other tasks for server maintenance.

3. Recommission the node.

4. Start all services components on the node.

5. Run Fsck utility for HDFS to ensure that HDFS is in healthy state. FSCK reports usually show a few over replicated blocks after a datanode is recommissioned which would automatically be fixed over time.

[#2]

You can do hotswap introduced by HDFS-1362 to replace slave nodes disks without Decommission/Recommission(Restart).

Ambari may not support this now. But you can always do that with hdfs command line.

More details can be found from this link.

[#3]

If I don't mind some down time, can I skip decommission / recommission process?

I have 12 DataNode locations and 12 disks. I would like to replace only one disk. I can schedule maintenance window.