I would like to ask for a little help on an error we are facing in the our HDP 188.8.131.52 Production Cluster with HBase version 1.1.2 and Phoenix version: 4.7.
Phoenix queries on HBase throws below exceptions.
INFO [hconnection-0x60129b9a-shared--pool1-t209] client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=68365 ms ago, cancelled=false, msg=row 'ï¿½aï¿½ï¿½ï¿½ï¿½ï¿½<MFI0000001' on table 'NS1:CUST_DETAILS' at region=NS1:CUST_DETAILS,,1574367014354.7ff0079c5f5238b8fc11f59ca97a73a3., hostname=datanode02.domain.com,16020,1581716720689, seqNum=18396
After narrowing down the issue we found mismatch row count between HBase and Phoenix. And as a workaround, deleting the entries for that particular table from system.stats table resolved the RpcRetryingCaller Issue temporarily as suggested on multiple blogs and communities . But after some time the issue occurs again. Also there are inconsistencies in results while running conditional queries because of this.
Is there any permanent solution for this? Can anyone help us to resolve this issue?