question Re: HDFS - Under-Replicated Blocks, missing Blocks in Archives of Support Questions (Read Only)

HDFS - Under-Replicated Blocks, missing Blocks

ANKITINTEL — Fri, 16 Sep 2022 09:48:55 GMT

my four node (1 Name , 3 Data node) cluster just recoverd from Hardrvie failure; I hade to re-formate three of data node drives ( all on node 4)and re mount it to designated location.

CDH manager shows following health issues and Impala , Hbase services are stopped.

hard drive failure was in node4, node 1 is master.

HDFS shows underreplicated blocks (total 1 block is missing and around 99% are under replicated);

I wonder how can I solve these issues. From my opinion hdfs should re-generate undereplicated blocks itself; if yes how long should it take? is there a way to experdite this process (make it faster)?

if No how can I manually re-generate such blocks?

thanks

Re: HDFS - Under-Replicated Blocks, missing Blocks

ANKITINTEL — Tue, 10 Nov 2015 17:35:38 GMT

following steps worked for me

in my case the node (#4) had disk failures was not coming up as data node ; was stopped
i tried starting it thougth services>hdfs>node4>start this data node but failed
while looking at logs o found warnings about more than one drives having data inconcistency; theere were FATAL messages but those lead me to dead end while searching for solution.
i went to all those mounted points(disks) and removed dfs by rm -rf /dfs on node; one of drive was unable to mount beacuse on that local location hdfs already made those folders; due to pervious attempts of starting datanode; i removed those folders as well and was able to mount. once mounted the new disk was clean free of any hadoop folders
I found a file in /tmp/hsprfdata_hdfs which began with some numbers (was some kind of log); i renamed it as old_**** just in case I need it later; and let hdfs know that file is gone!!!
i went back to CDH manager HDFS node 4 and tried restarting this datanode; it worked; all the disks have dfs and directories now; tmp location had that old_*** file removed and there is new file (some other number)

Re: HDFS - Under-Replicated Blocks, missing Blocks

cjervis — Tue, 10 Nov 2015 19:47:16 GMT

Congratulations on solving the issue. Thank you for sharing the steps in case someone else faces something similar.