question Re: One Datanode has the status DEAD in Support Questions

One Datanode has the status DEAD

Koffi — Tue, 08 Feb 2022 20:52:05 GMT

Hi we have noticed through Ambari that one of the datanode has the status "DEAD"

It seems that the process to replicate the block from the datanode to the rest of the datanode has started. In the main time what should be the proper procedure to restart the dead datanode?

Thank you

Re: One Datanode has the status DEAD

Daming Xue — Wed, 09 Feb 2022 02:16:33 GMT

Hello

Firstly, identify the issues with the node, replace any faulty hardware if necessary

You can follow the below post for the procedure

https://community.cloudera.com/t5/Community-Articles/Adding-nodes-to-an-HDP-cluster/ta-p/245386

Re: One Datanode has the status DEAD

Koffi — Thu, 10 Feb 2022 18:25:32 GMT

Hi @Daming Xue ,

Thank you for your help. It seems that one of the disks located on the datanode host has a issue.

We have currently 6 workernode(datanodes) on our cluster ans each node has 6 attached disks(1 to 6) of 3T and 1 attached disk(7) of 1T. With one datanode dead we are now at 86% Disk usage(dfs used) with a good amount of blocks under replicated:

We are trying to increase the amount of space available by increasing the total space on the disk 7 dynamically from 1T to 3T. We are trying to do that before getting the datanode back up again. Do you know if it could have a impact if we proceed?

Thank you

Re: One Datanode has the status DEAD

Azhar_Shaikh — Thu, 17 Mar 2022 16:57:36 GMT

Hello @Koffi

Once the disk is dynamically increased (from 1T to 3T) there should be no impact and once added you can run hdfs balancer to balance the data across the datanodes.

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Re: One Datanode has the status DEAD

shehbazk — Thu, 17 Mar 2022 18:54:03 GMT

Hello @Koffi

The balancer will do the job for you, please refer to the below Official docs before configuring it.

1- Overview of the HDFS Balancer

2- Configuring the Balancer

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.