I have
seen few articles and questions on the community around Disaster Recovery.
However, its still not clear completely and hence posting a new question around
that:
As I understand, typically, these entities need to be
backed-up / synced between the clusters
Primary Datasets
HDFS Data Teeing - Flume /
Hortonworks Data Flow Copying / Replication -
distcp (invoking it manually), Falcon Hive Data Behind the scenes, Hive
data is stored in HDFS. So I presume the techniques of teeing / copying
can be employed for HDFS as above can be used here as well. HBase Data HBase native DR
replication mechanism - master-slave, master-master and cyclic (http://hbase.apache.org/book.html#_cluster_replication) Solr Indexes If indexes are being
stored in HDFS, HDFS techniques would cover Solr datasets as well DB backed services Hive Metadata - Periodic backup of the
database from primary to DR cluster Amber - Ambari DB contains
configurations for other ecosystem components. Periodic backup of the
database from primary to DR cluster Oozie - Oozie database contains
job and workflow level information. So this need to be backed up
regularly to the DR cluster Ranger - Ranger policy DB contains
info about the various policies impacting RBAC. Need to be backed up to
the DR cluster Configurations
Periodic backup of Ambari
Server and Agent configurations (Ambari folders under /etc and /var) Periodic backup of
Configuration files for each application or service under /etc directory Periodic backup of binaries
(/usr/hadoop/current) Periodic backup of any OS
specific changes at a node level in the primary cluster
Application / User data Queries on DR Strategy
Teeing vs Copying- Which one
is preferred over the other? Understand its scenario dependent. But which
has better adaptability and more widely used in the industry? Copying?
Is it necessary to have both
the main and the DR cluster on the same version of HDP? If not, what are
things to consider if same version is not possible?
Should it be like for like
topology between clusters in terms of component placement including
gateway nodes and zookeeper services?
How does security play out
for DR? Should both the cluster nodes be part of the same Kerberos realm
or can they be part of different realms?
Can the replication factor be
lower? Or it recommended to maintain it as the same as the primary
cluster?
Any specific network
requirements in terms of latency, speed etc. between the clusters
Is there a need to run
balancer on the DR cluster periodically?
How does encryption play out
between the primary and DR clusters? If encryption at rest is enabled in
the primary one, how is it handled in the DR cluster? What are the
implications of wire-encryption while transferring the data between the
clusters?
When HDFS snapshots is
enabled on the primary cluster, how does it work when data is being synced
to the DR cluster? Can Snapshots be exported onto another cluster? I
understand this is possible for HBase snapshots. But is it allowed in HDFS
case? For example, if a file is deleted on the primary cluster, but
available in the snapshot, will that be synced to the snapshot directory
on the DR cluster?
For services which involve
databases (Hive, Oozie, Ambari), instead of backing up periodically from
the primary cluster to the DR cluster, is it recommended to setup one HA
master in the DR cluster directly?
For configurations and
application data, instead of backing up at regular intervals, is there a
way to keep them in sync between the primary and DR clusters?
What extra / different
functionality will third party solutions like WANDisco provide in comparison to Falcon? I am trying to
understand the "active-active" working of WANDisco and why it is
not possible with Falcon.
What is the recommendation to
ensure gateway node services like Knox and client libraries are kept in
sync between the clusters?
What is the recommendation
for keeping application data, for example, Spark / Sqoop job level
information? Apologies for the lengthy post, but want to cover all the areas around DR. Hence posted in a single question. Thanks Vijay
... View more
