08-08-2017 05:31 AM
We are using cloudera director on AWS .For disaster recovery we are considering the below options .
1-Incase of volume failure- replacing the volume with the latest snapshot
2-Instance failure(retirement of instances etc)
Need steps to prepare a disaster recovery plan incase Cloudera director ec2 instance is impacted or retiring.
How to bring back the cloudera director on new ec2 instance ?
What all things to be considered during and before for recovering cloudera director.
08-08-2017 06:57 AM
(I replied in another thread as well. This repeats my response there, but adds more.)
If Director is down, then your clusters will still function normally. If you are using pay-as-you-go licensing (a.k.a. usage based billing), then Director should be kept running so that it can gather billing data. Otherwise, you can leave Director shut off unless you want to use it.
If you need to replace your Director instance, you can install Director on a new instance and then provide it access to the old Director's database. For MySQL, that involves the usual configuration for Director. For H2, you must copy the state.h2.db file, usually in /var/lib/cloudera-director-server/state.h2.db, from the old Director instance to the new one. For either scenario, you can also restore the database from a backup if necessary.
If you made any custom configuration changes to the old Director instance, in the application.properties file, then you should also carry them over to the new instance.
Director does not have a mechanism to automatically learn about clusters that are already in existence, either built by a prior Director instance or manually. That's one reason it's important to have database backups.
For DR of Director, do the following:
- Use MySQL for the Director database. Since you are on AWS, an RDS instance is ideal for this, since you get backups and failover along with it.
- Save regular snapshots of the Director instance. To be more focused, you can backup only the Director configuration files, such as application.properties. If you made changes to the configuration of any plugins, those need backing up as well. A snapshot of the entire Director instance is simplest, though.
- Save Director logs so that, after recovery, it is possible to investigate any failures that may have necessitated recovery.
If you know ahead of time that the Director instance is being retired, then you can detach the root EBS volume from it and attach it to a new instance, effectively moving Director to a new instance without having to create a fresh installation.