Hi, I am using HDP 2.6. I am trying to set HBase Incremental backup and restore procedure for our cluster and facing issue while taking incremental backup for 2nd time. Below is the procedure we followed and observations: Took HBase full backup of table T1 and restored it successfully in table T2. Added some records in T1 and took incremental backup. On restoring the backup in table T2, row count matches the T1 table i.e, restore is successful. Again added some records in T1 and took incremental backup. This time on restoring the backup, newly added records didn't get reflected in new table though the backup and restore commands are successful. It might be possible that backup not working and hence restore operation not reflecting the updates. Also tried flush and major_compact on the table but no luck. Below are commands used:   hbase backup create full hdfs://mycluster/tmp/hbase_backup/TEST/ TEST:T1 hbase backup create incremental hdfs://mycluster/tmp/hbase_backup/TEST/ TEST:T1 hbase restore /tmp/hbase_backup/TEST backup_1590913120689 -overwrite TEST:T1 TEST:T2   Restore Command Logs:   2020-05-31 13:52:04,420 INFO [main] impl.RestoreClientImpl: Restoring 'TEST:T1' to 'TEST:T2' from full backup image hdfs://mycluster/tmp/hbase_backup/TEST/backup_1590912526723/TEST/T1 2020-05-31 13:52:04,705 INFO [main] util.RestoreServerUtil: Truncating exising target table 'TEST:T2', preserving region splits 2020-05-31 13:52:09,001 INFO [main] hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 574423 for hbase on ha-hdfs:mycluster 2020-05-31 13:52:09,074 INFO [LoadIncrementalHFiles-0] hfile.CacheConfig: CacheConfig:disabled 2020-05-31 13:52:09,173 INFO [LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://mycluster/user/hbase/hbase-staging/restore/bd8df2d7e972675825a437932bacea62/0/f1d861e630f543798e343038d7affebf first=\x80\x00\x00\x00\x04\x1E\x03N last=\x80\x00\x00\x00*\x1Bp> 2020-05-31 13:52:09,417 INFO [main] hdfs.DFSClient: Cancelling HDFS_DELEGATION_TOKEN token 574423 for hbase on ha-hdfs:mycluster 2020-05-31 13:52:09,447 INFO [main] impl.RestoreClientImpl: Restoring 'TEST:T1' to 'TEST:T2' from file dirs: hdfs://mycluster/tmp/hbase_backup/TEST//backup_1590912582304/TEST/T1//data,hdfs://mycluster/tmp/hbase_backup/TEST//backup_1590913120689/TEST/T1//data 2020-05-31 13:52:09,646 INFO [main] mapreduce.MapReduceRestoreService: Restore incremental backup from directory /user/hbase/hbase-staging/restore/TEST/T1,/user/hbase/hbase-staging/restore/TEST/T1 from hbase tables ,TEST:T1 to tables ,TEST:T2 2020-05-31 13:52:09,646 INFO [main] mapreduce.MapReduceRestoreService: Restore TEST:T1 into TEST:T2 2020-05-31 13:52:09,771 WARN [main] mapreduce.LoadIncrementalHFiles: Skipping non-directory hdfs://mycluster/user/hbase/hbase-staging/restore/TEST/T1/_SUCCESS 2020-05-31 13:52:09,773 WARN [main] mapreduce.LoadIncrementalHFiles: Skipping non-file FileStatus{path=hdfs://mycluster/user/hbase/hbase-staging/restore/TEST/T1/data/0; isDirectory=true; modification_time=1590913329548; access_time=0; owner=hbase; group=hdfs; permission=rwxr-xr-x; isSymlink=false} 2020-05-31 13:52:09,793 WARN [main] mapreduce.LoadIncrementalHFiles: Bulk load operation did not find any files to load in directory /user/hbase/hbase-staging/restore/TEST/T1. Does it contain files in subdirectories that correspond to column family names? 2020-05-31 13:52:09,889 WARN [main] mapreduce.LoadIncrementalHFiles: Skipping non-directory hdfs://mycluster/user/hbase/hbase-staging/restore/TEST/T1/_SUCCESS 2020-05-31 13:52:09,892 WARN [main] mapreduce.LoadIncrementalHFiles: Skipping non-file FileStatus{path=hdfs://mycluster/user/hbase/hbase-staging/restore/TEST/T1/data/0; isDirectory=true; modification_time=1590913329548; access_time=0; owner=hbase; group=hdfs; permission=rwxr-xr-x; isSymlink=false} 2020-05-31 13:52:09,912 WARN [main] mapreduce.LoadIncrementalHFiles: Bulk load operation did not find any files to load in directory /user/hbase/hbase-staging/restore/TEST/T1. Does it contain files in subdirectories that correspond to column family names? 2020-05-31 13:52:09,940 INFO [main] impl.RestoreClientImpl: TEST:T1 has been successfully restored to TEST:T2 2020-05-31 13:52:09,940 INFO [main] impl.RestoreClientImpl: Restore includes the following image(s): 2020-05-31 13:52:09,940 INFO [main] impl.RestoreClientImpl: Backup: backup_1590912526723 hdfs://mycluster/tmp/hbase_backup/TEST//backup_1590912526723/TEST/T1/ 2020-05-31 13:52:09,940 INFO [main] impl.RestoreClientImpl: Backup: backup_1590912582304 hdfs://mycluster/tmp/hbase_backup/TEST//backup_1590912582304/TEST/T1/ 2020-05-31 13:52:09,940 INFO [main] impl.RestoreClientImpl: Backup: backup_1590913120689 hdfs://mycluster/tmp/hbase_backup/TEST//backup_1590913120689/TEST/T1/ 2020-05-31 13:52:10,077 INFO [main] impl.RestoreClientImpl: Restore for [TEST:T1] are successful!   Any suggestions on what is going wrong.  Thanks! ... View more

Hi, I would like to ask for a little help on an error we are facing in the our HDP 2.6.0.1 Production Cluster with  HBase version 1.1.2 and  Phoenix version: 4.7.   Phoenix queries on HBase throws below exceptions. INFO  [hconnection-0x60129b9a-shared--pool1-t209] client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=68365 ms ago, cancelled=false, msg=row '�a�����<MFI0000001' on table 'NS1:CUST_DETAILS' at region=NS1:CUST_DETAILS,,1574367014354.7ff0079c5f5238b8fc11f59ca97a73a3., hostname=datanode02.domain.com,16020,1581716720689, seqNum=18396 After narrowing down the issue we found mismatch row count between HBase and Phoenix. And as a workaround, deleting the entries for that particular table from system.stats table resolved the RpcRetryingCaller Issue temporarily as suggested on multiple blogs and communities . But after some time the issue occurs again. Also there are inconsistencies in results while running conditional queries because of this. Is there any permanent solution for this? Can anyone help us to resolve this issue?   Thanks in Advance! ... View more

Hi Team,   I ran HBase hbck command and found below inconsistency.   ERROR: Region { meta => null, hdfs => null, deployed => hdpdatanode48.example.com,16020,1579287532060;CONTACTS:PHONE_DETAILS,\x80\x00\x00\x00\x1E\xD5\xEBW\x80\x00\x00\x00]\x08#\xBEMFI0000015,1581084039611.ccdd8d71cc8f081595c8cc01c4cf233f., replicaId => 0 }, key=ccdd8d71cc8f081595c8cc01c4cf233f, not on HDFS or in hbase:meta but deployed on hdpdatanode48.example.com,16020,1579287532060   The region "ccdd8d71cc8f081595c8cc01c4cf233f" can be seen on HBase-Master UI on the specified region-server, but this region is not recognized by any HBase command (ex: assign).   Compaction on this region is stuck from last few days and because of that no other regions on that region-server are being compacted causing the store-files to increase large in number.  Tried to repair the table but didn't worked.   How can I remove this inconsistency. ... View more

Hi, I would like to ask for a little help on an error we are facing in the newly setup kerberized HDP 2.6.0.1 cluster with 3 DN and 3 RS.  For the table with 850000 records, below select query upto 550000 records using phoenix works fine. select * from NS1:PHONE_DETAILS limit 550000; But on increasing the select count, it throws below exception. INFO  [hconnection-0x60129b9a-shared--pool1-t209] client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=68365 ms ago, cancelled=false, msg=row '�a�����<MFI0000001' on table 'NS1:PHONE_DETAILS' at region=NS1:PHONE_DETAILS,,1574367014354.7ff0079c5f5238b8fc11f59ca97a73a3., hostname=datanode02.domain.com,16020,1581716720689, seqNum=18396 Thanks in advance! ... View more

We are experiencing very high network usage on our HDP 2.6.0.1 Production Cluster with 52 RegionServers. Everything was working fine and suddenly rise in HBase CallQueue and Request Handlers is observed along with high Network Usage. Usually HBase Request count is in 300k to 500k which suddenly dropped below 10k. Tried many solutions but the issue still persists. Any suggestion would be very helpful. Thanks in Advance! ... View more