Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to properly migrate data (HDFS+HBase) from an existing cluster to a new one

Solved Go to solution
Highlighted

How to properly migrate data (HDFS+HBase) from an existing cluster to a new one

New Contributor

Hi, Folks.
We have a Hadoop cluster with ~1.5 PB of data (i.e. ~1500 TB), running on bare metal with CDH 5.7 and without Cloudera Manager. We're planning to decommission the cluster and set up a new one from the scratch (bare metal as well, not cloud), problably switching to Hortonworks (HDP) this time. We're also moving the whole datacenter where it's currently located, so the new one will be on a different location. The idea is to keep all the data (all 1.5 PB of data is relevant, so unfortunately we can't get rid of anything). Just to clarify, we're talking about HDFS data as well as HBase databases/tables.

That being said, my question is:
Assuming we have our brand-new cluster set up and ready to ingest the data, what would be the best method to migrate all 1.5 PB of it to the new one? Needless to say we need to have the least possible downtime while doing all this.

Below is our current cluster's resources:

  • 2 NameNodes in HA --> 2.80GHz 6-core / 24GB RAM
  • 49 DataNodes:
    • 5 of them --> 2.4GHz 6-cores / 72GB RAM
    • 38 of them --> 2.3GHz 16-cores / 128GB RAM
    • 6 of them --> 2.4GHz 32-cores / 128GB RAM


Thanks in advance!

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to properly migrate data (HDFS+HBase) from an existing cluster to a new one

While I don't want to oversimplify this process nor not suggest that Hortonworks Professional Services doesn't do these conversions with customers all the time (there is often more at play than simply moving the data, such as testing apps before & after), but... you can leverage DistCp, https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html, as your tool to move the data from your original cluster to your new one.

For the HBase data, I'd look to its Snapshots feature, http://hbase.apache.org/book.html#ops.snapshots, including its ability to export the snapshot to another cluster, as a solid approach.

Good luck and happy Hadooping!

View solution in original post

3 REPLIES 3
Highlighted

Re: How to properly migrate data (HDFS+HBase) from an existing cluster to a new one

While I don't want to oversimplify this process nor not suggest that Hortonworks Professional Services doesn't do these conversions with customers all the time (there is often more at play than simply moving the data, such as testing apps before & after), but... you can leverage DistCp, https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html, as your tool to move the data from your original cluster to your new one.

For the HBase data, I'd look to its Snapshots feature, http://hbase.apache.org/book.html#ops.snapshots, including its ability to export the snapshot to another cluster, as a solid approach.

Good luck and happy Hadooping!

View solution in original post

Highlighted

Re: How to properly migrate data (HDFS+HBase) from an existing cluster to a new one

New Contributor

Hi, Lester.

Thanks for your response. I didn't know about HBase's snapshot feature. I'll dig into it.
Regarding distcp, I was also thinking about using it, although I'm not sure how much time it will take to copy all the data, but I'll definitely check on it as well.


Best regards,
Juan

Re: How to properly migrate data (HDFS+HBase) from an existing cluster to a new one

Contributor

hey @Juan Castrilli let the community know how did you proceed :)

Don't have an account?
Coming from Hortonworks? Activate your account here