Teeing vs Copying- Which one is preferred over the other? Understand its scenario dependent. But which has better adaptability and more widely used in the industry? Copying?
With Teeing, you can split up primary tasks between the 2 clusters and use the other cluster as DR for that task. As an example, if you have clusters C1 and C2, you can use C1 as primary cluster and C2 as DR for some teams/tasks and use C2 as primary cluster and C1 as DR for some other users/tasks
Is it necessary to have both the main and the DR cluster on the same version of HDP? If not, what are things to consider if same version is not possible?
It is convinent to have them both on same version. This is especially the case if you want to use DR with almost no code changes if primary server is down.
Should it be like for like topology between clusters in terms of component placement including gateway nodes and zookeeper services? This is not required.
How does security play out for DR? Should both the cluster nodes be part of the same Kerberos realm or can they be part of different realms?
As a DR, same realm is a lot easier to manage than cross realm. But cross realm is possible.
Can the replication factor be lower? Or it recommended to maintain it as the same as the primary cluster? I have seen using rep factor 2 on DR clusters, but in case this turns in primary after disaster you may have to change rep factor to 3 on all data sets.
Any specific network requirements in terms of latency, speed etc. between the clusters
For ditscp, each node one cluster should communicate with each of the other nodes on second cluster.
Is there a need to run balancer on the DR cluster periodically?
Yes. Always good to run balancer to keep similar number of blocks across nodes.
How does encryption play out between the primary and DR clusters? If encryption at rest is enabled in the primary one, how is it handled in the DR cluster? What are the implications of wire-encryption while transferring the data between the clusters? Wire encyprtion will slow down transfers a little bit.
When HDFS snapshots is enabled on the primary cluster, how does it work when data is being synced to the DR cluster? Can Snapshots be exported onto another cluster? I understand this is possible for HBase snapshots. But is it allowed in HDFS case? For example, if a file is deleted on the primary cluster, but available in the snapshot, will that be synced to the snapshot directory on the DR cluster? If you are using snapshots, you can simply use distcp on snapshots instead of actual data set.
For services which involve databases (Hive, Oozie, Ambari), instead of backing up periodically from the primary cluster to the DR cluster, is it recommended to setup one HA master in the DR cluster directly? I don't think automating ambari is a good idea. Configs don't change that much so a simple process of duplicating might be better. Backing up would mean you need to have same hostnames and same topology.
For hive, instead of complete backup, Falcon can take care of table level replication.
For configurations and application data, instead of backing up at regular intervals, is there a way to keep them in sync between the primary and DR clusters?
Not sure where your application data resides, but for configuration since everything is managed by ambari, you can need to keep ambari configuration in sync.
... View more
