question Re: How many NameNodes needed on Backup Cluster in Support Questions

How many NameNodes needed on Backup Cluster

mhendricks — Thu, 11 Feb 2016 11:37:16 GMT

The customer wants to setup a backup to a backup cluster. They want to know how many NameNodes are required on the Backup Cluster. They want to know if there is some rule of thumb. The constraints for the backup cluster is below

Constraints
- Backup window: 1am-5am ET (12am-4am CT)
- Want to have a yarn job that runs Falcon backup jobs in another queue when the cluster isn’t busy - that way it doesn't pile up for the night.
- The clusters will be in Amazon, so they want the backup cluster to be backed by Amazon S3 storage (being used as HDFS)

Re: How many NameNodes needed on Backup Cluster

nsabharwal — Thu, 11 Feb 2016 11:40:39 GMT

@mhendricks

2 namenode

Active and passive setup

I would treat backup cluster setup "not config" almost same as production

Configs = CPU & Memory

Re: How many NameNodes needed on Backup Cluster

nsabharwal — Thu, 11 Feb 2016 11:47:48 GMT

@mhendricks

The overall flows looks good.

Re: How many NameNodes needed on Backup Cluster

jstraub — Thu, 11 Feb 2016 14:12:21 GMT

Hi @mhendricks you should use 2 Namenodes to increase the availability of the backup cluster. This is especially important if you only have limited window for your backup (like 1-5am).

This might also be helpful:

https://community.hortonworks.com/questions/4135/what-to-backup-and-how-only-metadata-not-data.html

https://community.hortonworks.com/questions/394/what-are-best-practices-for-setting-up-backup-and.html

Re: How many NameNodes needed on Backup Cluster

aervits — Thu, 11 Feb 2016 16:28:06 GMT

+1 cost savings are great but not with backups