One Datanode has the status DEAD

[#1]

Hi we have noticed through Ambari that one of the datanode has the status "DEAD"

It seems that the process to replicate the block from the datanode to the rest of the datanode has started. In the main time what should be the proper procedure to restart the dead datanode?

Thank you

[#2]

Hello

Firstly, identify the issues with the node, replace any faulty hardware if necessary

You can follow the below post for the procedure 

https://community.cloudera.com/t5/Community-Articles/Adding-nodes-to-an-HDP-cluster/ta-p/245386

[#3]

Hi @Daming Xue ,

Thank you for your help. It seems that one of the disks located on the datanode host has a issue.

 We have currently 6 workernode(datanodes) on our cluster ans each node has 6 attached disks(1 to 6) of 3T and 1 attached disk(7) of 1T. With one datanode dead we are now at 86% Disk usage(dfs used) with a good amount of blocks under replicated:

We are trying to increase the amount of space available by increasing the total space on the disk 7 dynamically from 1T to 3T. We are trying to do that before getting the datanode back up again. Do you know if it could have a impact if we proceed? 

Thank you 

[#4 · Accepted solution]

Hello @Koffi 

Once the disk is dynamically increased (from 1T to 3T) there should be no impact and once added you can run hdfs balancer to balance the data across the datanodes. 

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

[#5]

Hello @Koffi 

The balancer will do the job for you, please refer to the below Official docs before configuring it.

1- Overview of the HDFS Balancer 

2- Configuring the Balancer 

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.