Support Questions
Find answers, ask questions, and share your expertise

Query on Backup and DR solution for HDFS Data

Explorer

Hi Everyone,

 

Wanted to check your opinion on one backup and DR situation. So for one of our Dev Platform, we are still on CDH 5.4 version (with EOD already passed not under technical support scope), all the servers for this platform are going to be physically migrated from one Data Center to another Data Center. These 34 servers are placed in three separate racks, so considering in-built replication and Rack awareness, so technically I believe as long as we loose few servers and even one single Rack, we should still not be facing any data loss. Now considering the worst case scenario that multiple Racks or all servers are damaged during migration, I need to have a backup and DR solution in place.

 

To consider few limitations, we don’t have architectural capacity to spin up separate cluster to copy existing hdfs data there for recovery purpose. Currently we are thinking to backup all servers using Avamar backup tool, all OS data and data disks we can backup. I am going to take Namenode metadata backup, and backend database backup for applicable services separately as well.

 

My confusion is about HDFS data recovery, so let’s say if for all data nodes we are taking backup of all DataNode directories (all hard disks) and in case something catastrophic happens during migration and we have to rebuild the cluster again, considering we have backup of namenode metadata and all DataNode directories(block-pool structure in each DataNode), we should be able to completely recover the HDFS data, is this understanding correct? Or am I missing some technical details here?

 

Also, what other options we can use to plan DR around HDFS data?

1 ACCEPTED SOLUTION

Accepted Solutions

Expert Contributor

There are several options for you here, you have to pick one that best matches your requirements. If you are looking to backup your HDFS data but don't have capacity for a DR cluster, then backup to AWS S3 may be an option for you. Requires a cluster upgrade to later 5.x version, though, as this option is not there on 5.4 version.

 

Besides that, if you are able to completely restore the cluster nodes from your filesystem backups then you will be able to restore your HDFS data. 

View solution in original post

1 REPLY 1

Expert Contributor

There are several options for you here, you have to pick one that best matches your requirements. If you are looking to backup your HDFS data but don't have capacity for a DR cluster, then backup to AWS S3 may be an option for you. Requires a cluster upgrade to later 5.x version, though, as this option is not there on 5.4 version.

 

Besides that, if you are able to completely restore the cluster nodes from your filesystem backups then you will be able to restore your HDFS data. 

View solution in original post