Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HBase table counts mismatch after restoring HBase table snapshots

avatar
Rising Star

Hello Team,

 

I am facing issue with HBase snapshots restoring for big table in one of my environment but the same snapshot is working fine in other environment.

 

Created snapshot in X environment and exported to other Y environment, Once exported we are restoring the table in Y environment. So, here the problem is we are seeing counts mismatch and seeing more rows in Y when comparing with X.

 

NOTE1: Here there is no r/w operations when we create a snapshots in X environment.

NOTE2: There is no r/w operations in Y when we are restoring the table.

 

But the same snapshot when we truncate the table and restore, it is working fine in Y, I dont see any issues in HBase logs aswell.

Can someone please help me to solve this issue.

 

Thanks,

Vinod

2 REPLIES 2

avatar
Expert Contributor

@kvinod 

 

Can you provide the procedure and exact command you are using to restore the snapshot?

 

1) Are you restoring the snapshot over the top of Y environment, or are you clearing it first? This kind of behavior often happens when a restoration does not overwrite existing content, merely adding to it.

 

2) Are the versions exactly the same between the two environments? It is sometimes necessary to modify the command and import a different version, due to subtle differences between them.

 

If you could tell us your version information, that would also be useful. For example, there is a default method from CDH5:

 

https://my.cloudera.com/knowledge/Copying-HBase-Table-Between-Clusters--ExportSnapshot?id=72706

 

But there is a problem with it that appears in some CDH6 versions:

 

https://my.cloudera.com/knowledge/TSB-2020-379-Data-loss-with-restore-snapshot?id=283633

 

Regards,

Ryan Blough, COE

Cloudera Inc.

avatar
Rising Star

Hello @rblough 

 

First of all thank you so much for your response..!!

 

1. The X and Y environments are having different HBase versions.

2. HBase X version is 1.3.1 and HBase Y Version is 1.0.0

3. We are not clearing the table but following below steps,

 

echo "disable '$1'" | hbase shell
echo "restore_snapshot '$1_SNAPSHOT_$DATE'" | hbase shell
echo "enable '$1'" | hbase shell

 

Daily we are doing the same but sometimes we are seeing the counts mismatch.

So, we are doing truncating and restoring the same snapshots and working fine in Y environment.

 

Please help us your inputs.

 

Thanks & Regards,

Vinod