question HDFS Data migration from one data center to other data center in Support Questions

HDFS Data migration from one data center to other data center

Visvanath_JP — Wed, 29 Sep 2021 10:43:46 GMT

what is the procedure to migrate HDFS DATA from one data centre to another data centre?

Re: HDFS Data migration from one data center to other data center

willx — Wed, 29 Sep 2021 16:14:31 GMT

The question could be more specific like what hadoop versions are two clusters, are both clusters secured, are they CDH/CDP or HDP. Do you only migrate data in HDFS layer or other layer, for example hive / hbase / kudu.

The most common way is using distcp to migrate data between hdfs clusters.

https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/scaling-namespaces/topics/hdfs-distcp-to-copy-files.html

If you are using CDH/CDP, BDR job is another choice (distcp integrated)

https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/replication-manager/topics/rm-dc-hdfs-replication.html

Distcp guide:

https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html#:~:text=DistCp%20(distributed%20copy)%20is%20a,specified%20in%20the%20source%20list.

Regards,

Will

If the answer helps, please accept as solution and click thumbs up.

Re: HDFS Data migration from one data center to other data center

Visvanath_JP — Wed, 29 Sep 2021 16:44:56 GMT

CDH Version 5.15.1

only we want to migrate HDFS data.

Re: HDFS Data migration from one data center to other data center

willx — Wed, 29 Sep 2021 16:50:38 GMT

Then above solutions meet your needs.