Copy one HBase table to another in a short duration

[#1]

I wanted to copy data from one HBase table to another, but it's taking Huuuuge time to do the same and meanwhile other jobs in cluster are failing due to same. I tried below 3 techniques and still time taken is on a higher side. Can you suggest some efficient approach for the same. 1) org.apache.hadoop.hbase.mapreduce.CopyTable 2)org.apache.hadoop.hbase.mapreduce.export 3)take snapshot of the table, then clone the snapshot into another table Thanks in advance..

[#2]

@Dheeraj,

Hbase snapshot is the best method for disaster, backup and recovery procedure,

snapshot 'sourceTable', 'sourceTable-snapshot'
clone_snapshot 'sourceTable-snapshot', 'newTable'

[#3]

Thanks Nitin, But I had already tried snapshot and clone and it's taking huuge time

[#4]

This is simplest and best method for this. But you can HTable API for backup as well,

HTable API (such as a custom Java application)

As is always the case with Hadoop, you can always write your own custom application that utilizes the publicAPI and queries the table directly. You can do this through MapReduce jobs in order to utilize that framework’s distributed batch processing advantages, or through any other means of your own design. However, this approach requires a deep understanding of Hadoop development and all the APIs and performance implications of using them in your production cluster.

[#5]

Are you copying data on same cluster or different cluster?

[#6]

Same cluster

[#7]

Then snapshot the best among them.

[#8]

I am sorry Nitin, don't have much understanding for HTableAPI, if you could suggest some different command please.

or is it possible to copy few data(say few columns ir rows) using snapshot so that I can copy in 4-5 iterations.

[#9]

How can we run in more than 1 iterations using snapshot pls

[#10]

@Dheeraj,

We cant run snapshot in multiple iterations but we can use copytable and copy data from one timestamp to other timestamp like,

http://hbase.apache.org/0.94/book/ops_mgt.html#copytable

CopyTable is a utility that can copy part or of all of a table, either to the same cluster or another cluster. The usage is as follows:

1. $ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable[--starttime=X][--endtime=Y][--new.name=NEW][--peer.adr=ADR] tablename