Skip to main content
Koffi
Koffi
Gold ⭐️
February 8, 2022
Solved

One Datanode has the status DEAD

Hi we have noticed through Ambari that one of the datanode has the status "DEAD"

It seems that the process to replicate the block from the datanode to the rest of the datanode has started. In the main time what should be the proper procedure to restart the dead datanode?

Koffi_0-1644353273327.png

 

Thank you

Azhar_Shaikh1
Azhar_Shaikh1Answer
Gold ⭐️
March 17, 2022

Hello @Koffi 

 

Once the disk is dynamically increased (from 1T to 3T) there should be no impact and once added you can run hdfs balancer to balance the data across the datanodes. 

 

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View in thread

4 replies

Daming Xue
Gold ⭐️
Daming Xue
Gold ⭐️
February 9, 2022

Hello

 

Firstly, identify the issues with the node, replace any faulty hardware if necessary

You can follow the below post for the procedure 

https://community.cloudera.com/t5/Community-Articles/Adding-nodes-to-an-HDP-cluster/ta-p/245386

Translate
Koffi
KoffiAuthor
Gold ⭐️
February 10, 2022

Hi @Daming Xue ,

Thank you for your help. It seems that one of the disks located on the datanode host has a issue.

 We have currently 6 workernode(datanodes) on our cluster ans each node has 6 attached disks(1 to 6) of 3T and 1 attached disk(7) of 1T. With one datanode dead we are now at 86% Disk usage(dfs used) with a good amount of blocks under replicated:

Koffi_0-1644517196817.png

 

We are trying to increase the amount of space available by increasing the total space on the disk 7 dynamically from 1T to 3T. We are trying to do that before getting the datanode back up again. Do you know if it could have a impact if we proceed? 

 

Thank you 

Translate
Azhar_Shaikh1
Gold ⭐️
Azhar_Shaikh1Answer
Gold ⭐️
March 17, 2022

Hello @Koffi 

 

Once the disk is dynamically increased (from 1T to 3T) there should be no impact and once added you can run hdfs balancer to balance the data across the datanodes. 

 

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Translate
shehbazk
Gold ⭐️⭐️
shehbazk
Gold ⭐️⭐️
March 17, 2022

Hello @Koffi 

 

The balancer will do the job for you, please refer to the below Official docs before configuring it.

 

1- Overview of the HDFS Balancer 

2- Configuring the Balancer 

 

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Translate
Terms & ConditionsAccessibility statement