Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Import and export hbase snapshot from shared location

Highlighted

Import and export hbase snapshot from shared location

Contributor

Hi all,

 

      With the below steps we are able to copy the hbase data from cluster to another cluster.

 

  1. snapshot "SOURCE_TABLE_NAME","SNAPSHOT_NAME"
  2. bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot SNAPSHOT_NAME -copy-to hdfs://DESTINATION_CLUSTER_ACTIVE_NAMENODE_ADDRESS:8020/hbase -mappers 16
  3. disable "DEST_TABLENAME"
  4. restore_snapshot "SNAPSHOT_NAME"

But the problem is Source cluster does not have access to the destination cluster namenode. To solve this we are planning to have one shared location which mounted in both cluster nodes and using those we are planing to do import and export the snapshot to avoid multiple time copy of the files since the hbase table data size is hug which is around 10TB of data.

 

Can you please tell me the command to 

  1. export the snapshot to the local mounted shared File location (instead of hdfs location)
  2. restore snapshot from the local  mounted shared file location. (instead of hdfs location)

Thank you,

J.Ganesh Kumar

1 REPLY 1

Re: Import and export hbase snapshot from shared location

Master Guru
If speed is key here, look at getting the connectivity setup between the clusters, as otherwise you're headed down routes of adding duplicate data copy stages.

You can pull down a HDFS export location to disk via a regular 'hadoop fs -put' or via a local 'hadoop distcp' job with a file:///path destination (ensure you don't overwhelm the file-service if this is NFS/etc. by keeping the mapper count low). This will still be slower than just directly sending it to the other cluster's HDFS.