Hbase Phoenix Jobs failing with "WrongRegionException: Requested row out of range for Get on HRegion"

[#1]

Hi Team,

We are using Jmeter to submit job (1300/hr) to hbase/phoenix. HDP3.1.4 and Phoenix 5.0

Job starts failing with below error -

2022-08-25 16:21:44,785 INFO org.apache.phoenix.iterate.BaseResultIterators: Failed to execute task during cancel
java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.exceptions.ScannerResetException: Scanner is closed on the server-side
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3468)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for Get on HRegion OBST:DOCUMENT_METADATA,\x0C\x00\x00\x00,1659594973530.146ed04497483dae508d10d1e2676a12., startKey='\x0C\x00\x00\x00', getEndKey()='\x0CADELMWSQRP\x004bcdbe31987c05d9e88cba377df31f3bbaae274d7df670ed26690fb021c90f5b\x00PERSISTENT', row='\x0CADELSRD\x009bb7104f2f156cec8ecb0e53f95b72affa43969125732ab898c96282356999f7\x00PERSISTENT'
at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5713)
at org.apache.hadoop.hbase.regionserver.HRegion.prepareGet(HRegion.java:7297)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7290)
at org.apache.phoenix.util.IndexUtil.wrapResultUsingOffset(IndexUtil.java:514)
at org.apache.phoenix.iterate.RegionScannerFactory$1.nextRaw(RegionScannerFactory.java:197)
at org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:77)
at org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:77)
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:274)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3136)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3385)
... 5 more
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1439)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1352)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1239)
at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
at org.apache.phoenix.iterate.LimitingResultIterator.next(LimitingResultIterator.java:47)
at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:805)
at org.apache.calcite.avatica.jdbc.JdbcResultSet.frame(JdbcResultSet.java:148)
at org.apache.calcite.avatica.jdbc.JdbcResultSet.create(JdbcResultSet.java:101)
at org.apache.calcite.avatica.jdbc.JdbcMeta.execute(JdbcMeta.java:887)
at org.apache.calcite.avatica.remote.LocalService.apply(LocalService.java:254)
at org.apache.calcite.avatica.remote.Service$ExecuteRequest.accept(Service.java:1032)
at org.apache.calcite.avatica.remote.Service$ExecuteRequest.accept(Service.java:1002)
at org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:94)
at org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46)
at org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:127)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.apache.phoenix.shaded.org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.apache.phoenix.shaded.org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.exceptions.ScannerResetException: Scanner is closed on the server-side
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3468)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for Get on HRegion OBST:DOCUMENT_METADATA,\x0C\x00\x00\x00,1659594973530.146ed04497483dae508d10d1e2676a12., startKey='\x0C\x00\x00\x00', getEndKey()='\x0CADELMWSQRP\x004bcdbe31987c05d9e88cba377df31f3bbaae274d7df670ed26690fb021c90f5b\x00PERSISTENT', row='\x0CADELSRD\x009bb7104f2f156cec8ecb0e53f95b72affa43969125732ab898c96282356999f7\x00PERSISTENT'

Same time we tried to check "select count(*)" with and without index but it gives difference as shown below -

NOTE: Below output is from test cluster where we were able to repro issue. View name might differ in below screenshot -

We suspected below apache bug for  "WrongRegionException: Requested row out of range for Get on HRegion" - 

https://issues.apache.org/jira/browse/PHOENIX-3828

For "select count(*)" mismatch - we suspected we are hitting - 
[PHOENIX-6090] Local indexes get out of sync after changes for global consistent indexes - ASF JIRA ...

Can someone help on debugging steps?