Skip to main content
raniaa
raniaa
Silver ⭐️⭐️
December 21, 2021
Question

HDFS blocks missing at every restart of CM cluster

Hi,

I am facing issues with HDFS in my Cloudera Manager cluster. I have a cluster of 4 virtual machines (1 master and 3 slaves) on cloud and at every shutdown and restart of the cluster/vms HDFS shows some missing blocks.

I have found a "workaround" as deleting the missing files and restarting the service would solve the problem but sometimes the missing files contain important data related to other hadoop services that generate issues if they are removed. Is there another solution to avoid deleting data ?

Thank you.

5 replies

Amithsha
Gold ⭐️
Amithsha
Gold ⭐️
January 16, 2022

Missing blocks can happen because of 2 reasons,

1. Data corruption - Either the disk is corrupted or the VM is down. This is a permanently failure means the data is lost.

2. Delay is Datanode report - This is because of a delay in datanode report and this is a temporary failure and in some time if datanode reports back the report then the cluster will be back to normal.


How to find and solve the issue. 

1. Make sure no data volume failures or VM failures.

2. Ensure that the namenode received heartbeats from all datanodes.
Namenode UI -> Datanodes -> last contact
If all datanodes found in the list and not in dead list or not found then no issues with receiving block report. 
Apart from this to debug further we need Namenode and datanode logs.

Translate
raniaa
raniaaAuthor
Silver ⭐️⭐️
January 18, 2022

Thank you for your response , so do you think that maybe because i shut down the virtual machines everyday the data is sometimes lost ? it is very unusual since the issue now is not happening every time but sometimes hdfs is corrupt and shows blocks missing and sometimes it's healthy from the start so i don't know what to think of it.

Translate
Amithsha
Gold ⭐️
Amithsha
Gold ⭐️
January 18, 2022

I suspect that your datanodes report is slow.Because after restart of namenode you are trigger the datanode restart so it will take time to come up with reports during that interval you can except for missing blocks this will be an intermediate issue. So that you can wait for few more min's and check the namenode ui. Else during the time of issue copy the logs and share it.

 

Make sure to mark the answer as the accepted solution. If it resolves your issue !

Translate
Meepoljd
Meepoljd
Gold ⭐️
January 17, 2022

Are you make sure your Datanodes are all started and working properly?When I restarted the cluster earlier,there are also many missingBlocks before the DataNode restarts completely.

Translate
Amithsha
Gold ⭐️
Amithsha
Gold ⭐️
January 17, 2022

When I restarted the cluster earlier,there are also many missingBlocks before the DataNode restarts completely.

This can cause the missing blocks.ex: Namenode got restarted but still the datanode restart is inprogress. So the heartbeat from datanode might be missed. To confirm that you can check the namenode UI post restart and during the time of missing blocks.

Translate
Terms & ConditionsAccessibility statement