Hadoop Cluster backup

[#1]

I have recreated all of our Hadoop cluster from scratch as the NN centos failed,

we had no backup so it was real disaster!

I would like to prevent this from happening again so please tell me what kind of backup do I need?

I have one master- cloudera manager and DB

Two NN

And 4 workers

Please let me know if you need more details...

[#2]

create master slave of postgrsql or mysql sync both machine NN,(highly available)
use cloudera manager on both NN, there will be no loose of data.
must create replicate factor at least 2.
if one namenode down another will be activate and mysql is sync automatically as i setup for my cluster if one node goes download servce switch to another no issue for me.

[#3]

@reutsc

You can enable "high availability". This is the normal practice in the industry for fault tolerance.

Note:

1. When you have high availability for NN. You have to keep two different NN in two different master. So that even one master is down, the other master will support without any interruption. 

2. One NN is active alway and other one is standby

3. Make sure both the masters have the same configuration, it will help NN to be in sync and few more advantages

4. High Availbility (HA) is not only for NN, there are few more other services will also support HA, you can refer the below link to know more

http://www.cloudera.com/documentation/cdh/5-0-x/CDH5-High-Availability-Guide/CDH5-High-Availability-...

http://www.cloudera.com/documentation/cdh/5-0-x/CDH5-High-Availability-Guide/cdh5hag_hdfs_ha_config....