question Re: What is a suggested offsite/cold backup method for HDFS? besides AWS S3 in Archives of Support Questions (Read Only)

What is a suggested offsite/cold backup method for HDFS? besides AWS S3

cspencer — Wed, 14 Oct 2015 05:07:33 GMT

Re: What is a suggested offsite/cold backup method for HDFS? besides AWS S3

emizell — Wed, 14 Oct 2015 22:39:30 GMT

Ideally, you don't need to backup HDFS since it stores 3 copies by default. If you need a DR strategy, a good strategy is to have a separate cluster in another datacenter. Use Apache Falcon or distcp to mirror the data to the DR cluster. If you need to backup certain high value datasets, take a snapshot of the data and back it up to tape (ugh!) or put it on your corporate SAN/NAS (if permitted). This will give you a way to recover the data if disaster strikes. I don't know if you are adverse to cloud storage (based on your S3 comment), but it is cheap and online all the time to recover data when needed.

I hope this helps,

Eric

Re: What is a suggested offsite/cold backup method for HDFS? besides AWS S3

sluangsay — Thu, 15 Oct 2015 00:56:29 GMT

"you don't need to backup HDFS since it stores 3 copies by default" : IMHO, I think we need to take care with that message. Having some replicas don't protect us again a "human error" or rogue administrator (hdfs dfs -rmr /), neither again an application bug.

It's just like RAID1: it's good but no IT department would consider that it serves as a backup.

Re: What is a suggested offsite/cold backup method for HDFS? besides AWS S3

nsabharwal — Thu, 15 Oct 2015 19:17:33 GMT

@Cassandra

HDFS Snapshots HBASE Snapshots Hive Metadata (DBA can work on setting up this based on DB flavor used for HCatalog)

Going back to your original question

This is helpful to understand the architecture. We can point it to DR cluster (It can be on prem or in cloud) as Eric mentioned.