HBase table counts mismatch after restoring HBase table snapshots

[#1]

Hello Team,

I am facing issue with HBase snapshots restoring for big table in one of my environment but the same snapshot is working fine in other environment.

Created snapshot in X environment and exported to other Y environment, Once exported we are restoring the table in Y environment. So, here the problem is we are seeing counts mismatch and seeing more rows in Y when comparing with X.

NOTE1: Here there is no r/w operations when we create a snapshots in X environment.

NOTE2: There is no r/w operations in Y when we are restoring the table.

But the same snapshot when we truncate the table and restore, it is working fine in Y, I dont see any issues in HBase logs aswell.

Can someone please help me to solve this issue.

Thanks,

Vinod

[#2]

@kvinod 

Can you provide the procedure and exact command you are using to restore the snapshot?

1) Are you restoring the snapshot over the top of Y environment, or are you clearing it first? This kind of behavior often happens when a restoration does not overwrite existing content, merely adding to it.

2) Are the versions exactly the same between the two environments? It is sometimes necessary to modify the command and import a different version, due to subtle differences between them.

If you could tell us your version information, that would also be useful. For example, there is a default method from CDH5:

https://my.cloudera.com/knowledge/Copying-HBase-Table-Between-Clusters--ExportSnapshot?id=72706

But there is a problem with it that appears in some CDH6 versions:

https://my.cloudera.com/knowledge/TSB-2020-379-Data-loss-with-restore-snapshot?id=283633

Regards,

Ryan Blough, COE

Cloudera Inc.

[#3]

Hello @rblough 

First of all thank you so much for your response..!!

1. The X and Y environments are having different HBase versions.

2. HBase X version is 1.3.1 and HBase Y Version is 1.0.0

3. We are not clearing the table but following below steps,

echo "disable '$1'" | hbase shell
echo "restore_snapshot '$1_SNAPSHOT_$DATE'" | hbase shell
echo "enable '$1'" | hbase shell

Daily we are doing the same but sometimes we are seeing the counts mismatch.

So, we are doing truncating and restoring the same snapshots and working fine in Y environment.

Please help us your inputs.

Thanks & Regards,

Vinod