Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

Highlighted

HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

Contributor

Hi All,

My cluster has been hacked and all files on HDFS been deleted. Is there any way I can recover the files that are "Pending Deletion Blocks" ?

Any help much appreciated

9 REPLIES 9
Highlighted

Re: HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

Contributor

is there any way of recovering the data from the underlying file system? i.e. /hadoop/hdfs/data/current ? I see there is files and folders in there.

Highlighted

Re: HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

Rising Star

@mike harding it is hard to do without very deep understanding of HDFS. if you have support from any vendor this would be the time to call for help, even with that you probably have suffered massive data loss. if you have trash enabled, you might have data inside it.

Highlighted

Re: HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

New Contributor

This won't help after the fact, but if your data is either hard or impossible to replace, you may want to consider backing up your data as a part of a larger DR program.

Highlighted

Re: HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

Super Guru
@mike harding

If you have support subscription, I would recommend you to log a support case with us.

FYI - https://community.hortonworks.com/articles/26181/how-to-recover-accidentally-deleted-file-in-hdfs.ht...

Re: HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

Contributor

It was around 4/5 hours after I found the deleted files. Would this still be an option?

Highlighted

Re: HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

Contributor

Thank you for all your suggestions. Just out of interest if i was able to to use extundelete and recover the block files and namenode files removed from /hadoop/data/namenode and /hadoop/hdfs/namenode, would this also be an option?

Highlighted

Re: HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

Contributor

p.s I don't have a subscription - i don't work for a company. I believe the delete happened at 1.07am and in the files in namenode I see there is two fsimage files. One at 1.07 and then another 6 hours later. Is there anyway I can roll back to the fsimage at 1.07am ? (see pic attached)

11218-output.png

Highlighted

Re: HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

Rising Star

@mike harding yes , you can. But in these six hours your cluster might have deleted all data blocks that needs to be deleted. So you might not be able to recover the data. Can you please post how many pending delete blocks are in the system. Please make sure that namenode remains in safe mode when you boot it up.

Highlighted

Re: HELP! My cluster has been hacked and all HDFS been deleted..says "HDFS Pending Deletion Blocks"..how can i recover?

Contributor

There is no pending delete blocks which would suggest the deletion has taken place. What I'm currently trying to do is use extundelete on /hadoop/hdfs/data to recover the files. If I can get the data blocks back, what would I have to do next to roll back to the checkpointed fsimage in namenode? Is it a simple case of deleting the latest fsimage and edits logs so HDFS boots from the previous fsimage file?

Any help much appreciated

Don't have an account?
Coming from Hortonworks? Activate your account here