Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

One Datanode has the status DEAD

avatar
Contributor

Hi we have noticed through Ambari that one of the datanode has the status "DEAD"

It seems that the process to replicate the block from the datanode to the rest of the datanode has started. In the main time what should be the proper procedure to restart the dead datanode?

Koffi_0-1644353273327.png

 

Thank you

1 ACCEPTED SOLUTION

avatar
Master Collaborator

Hello @Koffi 

 

Once the disk is dynamically increased (from 1T to 3T) there should be no impact and once added you can run hdfs balancer to balance the data across the datanodes. 

 

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

4 REPLIES 4

avatar
Expert Contributor

Hello

 

Firstly, identify the issues with the node, replace any faulty hardware if necessary

You can follow the below post for the procedure 

https://community.cloudera.com/t5/Community-Articles/Adding-nodes-to-an-HDP-cluster/ta-p/245386

avatar
Contributor

Hi @Daming Xue ,

Thank you for your help. It seems that one of the disks located on the datanode host has a issue.

 We have currently 6 workernode(datanodes) on our cluster ans each node has 6 attached disks(1 to 6) of 3T and 1 attached disk(7) of 1T. With one datanode dead we are now at 86% Disk usage(dfs used) with a good amount of blocks under replicated:

Koffi_0-1644517196817.png

 

We are trying to increase the amount of space available by increasing the total space on the disk 7 dynamically from 1T to 3T. We are trying to do that before getting the datanode back up again. Do you know if it could have a impact if we proceed? 

 

Thank you 

avatar
Master Collaborator

Hello @Koffi 

 

Once the disk is dynamically increased (from 1T to 3T) there should be no impact and once added you can run hdfs balancer to balance the data across the datanodes. 

 

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Master Collaborator

Hello @Koffi 

 

The balancer will do the job for you, please refer to the below Official docs before configuring it.

 

1- Overview of the HDFS Balancer 

2- Configuring the Balancer 

 

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.