Hi,
I am using HDP 2.6. I am trying to set HBase Incremental backup and restore procedure for our cluster and facing issue while taking incremental backup for 2nd time.
Below is the procedure we followed and observations:
- Took HBase full backup of table T1 and restored it successfully in table T2.
- Added some records in T1 and took incremental backup. On restoring the backup in table T2, row count matches the T1 table i.e, restore is successful.
- Again added some records in T1 and took incremental backup. This time on restoring the backup, newly added records didn't get reflected in new table though the backup and restore commands are successful.
It might be possible that backup not working and hence restore operation not reflecting the updates. Also tried flush and major_compact on the table but no luck.
Below are commands used:
hbase backup create full hdfs://mycluster/tmp/hbase_backup/TEST/ TEST:T1
hbase backup create incremental hdfs://mycluster/tmp/hbase_backup/TEST/ TEST:T1
hbase restore /tmp/hbase_backup/TEST backup_1590913120689 -overwrite TEST:T1 TEST:T2
Restore Command Logs:
2020-05-31 13:52:04,420 INFO [main] impl.RestoreClientImpl: Restoring 'TEST:T1' to 'TEST:T2' from full backup image hdfs://mycluster/tmp/hbase_backup/TEST/backup_1590912526723/TEST/T1
2020-05-31 13:52:04,705 INFO [main] util.RestoreServerUtil: Truncating exising target table 'TEST:T2', preserving region splits
2020-05-31 13:52:09,001 INFO [main] hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 574423 for hbase on ha-hdfs:mycluster
2020-05-31 13:52:09,074 INFO [LoadIncrementalHFiles-0] hfile.CacheConfig: CacheConfig:disabled
2020-05-31 13:52:09,173 INFO [LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://mycluster/user/hbase/hbase-staging/restore/bd8df2d7e972675825a437932bacea62/0/f1d861e630f543798e343038d7affebf first=\x80\x00\x00\x00\x04\x1E\x03N last=\x80\x00\x00\x00*\x1Bp>
2020-05-31 13:52:09,417 INFO [main] hdfs.DFSClient: Cancelling HDFS_DELEGATION_TOKEN token 574423 for hbase on ha-hdfs:mycluster
2020-05-31 13:52:09,447 INFO [main] impl.RestoreClientImpl: Restoring 'TEST:T1' to 'TEST:T2' from file dirs: hdfs://mycluster/tmp/hbase_backup/TEST//backup_1590912582304/TEST/T1//data,hdfs://mycluster/tmp/hbase_backup/TEST//backup_1590913120689/TEST/T1//data
2020-05-31 13:52:09,646 INFO [main] mapreduce.MapReduceRestoreService: Restore incremental backup from directory /user/hbase/hbase-staging/restore/TEST/T1,/user/hbase/hbase-staging/restore/TEST/T1 from hbase tables ,TEST:T1 to tables ,TEST:T2
2020-05-31 13:52:09,646 INFO [main] mapreduce.MapReduceRestoreService: Restore TEST:T1 into TEST:T2
2020-05-31 13:52:09,771 WARN [main] mapreduce.LoadIncrementalHFiles: Skipping non-directory hdfs://mycluster/user/hbase/hbase-staging/restore/TEST/T1/_SUCCESS
2020-05-31 13:52:09,773 WARN [main] mapreduce.LoadIncrementalHFiles: Skipping non-file FileStatus{path=hdfs://mycluster/user/hbase/hbase-staging/restore/TEST/T1/data/0; isDirectory=true; modification_time=1590913329548; access_time=0; owner=hbase; group=hdfs; permission=rwxr-xr-x; isSymlink=false}
2020-05-31 13:52:09,793 WARN [main] mapreduce.LoadIncrementalHFiles: Bulk load operation did not find any files to load in directory /user/hbase/hbase-staging/restore/TEST/T1. Does it contain files in subdirectories that correspond to column family names?
2020-05-31 13:52:09,889 WARN [main] mapreduce.LoadIncrementalHFiles: Skipping non-directory hdfs://mycluster/user/hbase/hbase-staging/restore/TEST/T1/_SUCCESS
2020-05-31 13:52:09,892 WARN [main] mapreduce.LoadIncrementalHFiles: Skipping non-file FileStatus{path=hdfs://mycluster/user/hbase/hbase-staging/restore/TEST/T1/data/0; isDirectory=true; modification_time=1590913329548; access_time=0; owner=hbase; group=hdfs; permission=rwxr-xr-x; isSymlink=false}
2020-05-31 13:52:09,912 WARN [main] mapreduce.LoadIncrementalHFiles: Bulk load operation did not find any files to load in directory /user/hbase/hbase-staging/restore/TEST/T1. Does it contain files in subdirectories that correspond to column family names?
2020-05-31 13:52:09,940 INFO [main] impl.RestoreClientImpl: TEST:T1 has been successfully restored to TEST:T2
2020-05-31 13:52:09,940 INFO [main] impl.RestoreClientImpl: Restore includes the following image(s):
2020-05-31 13:52:09,940 INFO [main] impl.RestoreClientImpl: Backup: backup_1590912526723 hdfs://mycluster/tmp/hbase_backup/TEST//backup_1590912526723/TEST/T1/
2020-05-31 13:52:09,940 INFO [main] impl.RestoreClientImpl: Backup: backup_1590912582304 hdfs://mycluster/tmp/hbase_backup/TEST//backup_1590912582304/TEST/T1/
2020-05-31 13:52:09,940 INFO [main] impl.RestoreClientImpl: Backup: backup_1590913120689 hdfs://mycluster/tmp/hbase_backup/TEST//backup_1590913120689/TEST/T1/
2020-05-31 13:52:10,077 INFO [main] impl.RestoreClientImpl: Restore for [TEST:T1] are successful!
Any suggestions on what is going wrong.
Thanks!
