See my answers inline below: --->So by replication do you mean distcp copy across two clusters - but distcp copies are not real time - so can they be called as replication? Yes, replication can be enabled by either distcp or if you are streaming data using some ingestion tool like Nifi, you can send data to both clusters (Active and backup) at the same time in real time. In practice and because of Physics, you cannot just have 100 percent sync between two clusters so distcp is almost always the way to go. --->I have read that Flume can be used to copy from a source to two different clusters - but even such a configuration would exist outside hdfs. Is this what you meant by replication? I would prefer Nifi over Flume, but by replication, my idea is more of using distcp. Two data centers may be 1000 miles apart cannot be in synced for every single second and the cost and complexity of trying to achieve that is also very high. I have seen some very low latency telco use cases and those are some very few use cases where active-active is justified (remember very complex). Highly recommend you read this. ---> I read that Hive can be backed up using Snapshots in an incremental way. That is take a snapshot of hive data at one point in time and from then on (to another point in time) take snapshots and use the difference feature to get the incrementals between the current snapshot and the previous one. So data can be recovered by loading the full Snapshot and the incrementals to a point in time (like RDBMS recovery). Is this workable? I am not aware of this method. There are two things you need to do for Hive replication. a. Replicate metadata (use MySQL replication techniques or whichever metadata database you are using) b. Use distcp to replicate HDFS files.
... View more
