What are the best practices for replicating HDFS and Hive data from a Production cluster to DR cluster?

[#1]

Can some one help me in what are the best practices and ways to do the Data replication in Prod cluster.

[#2]

Hi Suri,

@Suri Nuthalapati

Use falcon to mirror HDFS data.

http://hortonworks.com/hadoop-tutorial/mirroring-datasets-between-hadoop-clusters-with-apache-falcon...

Falcon can be used to mirror HIVE metadata.

http://hortonworks.com/hadoop-tutorial/mirroring-datasets-between-hadoop-clusters-with-apache-falcon...

Hope that helps!

[#3]

Hi Patel,

Thanks for your answer. Do you know any best practices for using Distcp. I am looking for best practices other than Falcon.

Thanks,

Suri

[#4]

Hi Suri,

With distcp you will not be able to replicate Hive metadata.Only HDFS data can be replicated!.

If you have any further question then feel free to ask.

If you like my answer then please select as best answer!

Thanks.

[#5]

@ Suri Nuthalapati

[#6]

Hive Metastore replication

https://cwiki.apache.org/confluence/display/Hive/Replication

https://cwiki.apache.org/confluence/display/Hive/HiveReplicationDevelopment

Once all the metastores are in HBase

https://hbase.apache.org/book.html#_cluster_replication

[#7]

Timothy, Thank you for your response. But I am looking for best ways to replicate HDFS also using Distcp.

Suri

[#8]

Distributed Copy (DistCP) http://hadoop.apache.org/docs/r2.7.3/hadoop-distcp/DistCp.html