Cloudera Community
sagarshimpi
user rank
Expert Contributor

Hbase Phoenix Jobs failing with "WrongRegionExcep...

by Expert Contributor in Support Questions
‎08-27-2022 03:01 PM
‎08-27-2022 03:01 PM
Hi Team,   We are using Jmeter to submit job (1300/hr) to hbase/phoenix. HDP3.1.4 and Phoenix 5.0 Job starts failing with below error - 2022-08-25 16:21:44,785 INFO org.apache.phoenix.iterate.BaseResultIterators: Failed to execute task during cancel java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.exceptions.ScannerResetException: Scanner is closed on the server-side at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3468) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for Get on HRegion OBST:DOCUMENT_METADATA,\x0C\x00\x00\x00,1659594973530.146ed04497483dae508d10d1e2676a12., startKey='\x0C\x00\x00\x00', getEndKey()='\x0CADELMWSQRP\x004bcdbe31987c05d9e88cba377df31f3bbaae274d7df670ed26690fb021c90f5b\x00PERSISTENT', row='\x0CADELSRD\x009bb7104f2f156cec8ecb0e53f95b72affa43969125732ab898c96282356999f7\x00PERSISTENT' at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5713) at org.apache.hadoop.hbase.regionserver.HRegion.prepareGet(HRegion.java:7297) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7290) at org.apache.phoenix.util.IndexUtil.wrapResultUsingOffset(IndexUtil.java:514) at org.apache.phoenix.iterate.RegionScannerFactory$1.nextRaw(RegionScannerFactory.java:197) at org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:77) at org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:77) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:274) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3136) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3385) ... 5 more at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1439) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1352) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1239) at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72) at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93) at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.iterate.LimitingResultIterator.next(LimitingResultIterator.java:47) at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:805) at org.apache.calcite.avatica.jdbc.JdbcResultSet.frame(JdbcResultSet.java:148) at org.apache.calcite.avatica.jdbc.JdbcResultSet.create(JdbcResultSet.java:101) at org.apache.calcite.avatica.jdbc.JdbcMeta.execute(JdbcMeta.java:887) at org.apache.calcite.avatica.remote.LocalService.apply(LocalService.java:254) at org.apache.calcite.avatica.remote.Service$ExecuteRequest.accept(Service.java:1032) at org.apache.calcite.avatica.remote.Service$ExecuteRequest.accept(Service.java:1002) at org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:94) at org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46) at org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:127) at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:539) at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.apache.phoenix.shaded.org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.apache.phoenix.shaded.org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.exceptions.ScannerResetException: Scanner is closed on the server-side at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3468) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for Get on HRegion OBST:DOCUMENT_METADATA,\x0C\x00\x00\x00,1659594973530.146ed04497483dae508d10d1e2676a12., startKey='\x0C\x00\x00\x00', getEndKey()='\x0CADELMWSQRP\x004bcdbe31987c05d9e88cba377df31f3bbaae274d7df670ed26690fb021c90f5b\x00PERSISTENT', row='\x0CADELSRD\x009bb7104f2f156cec8ecb0e53f95b72affa43969125732ab898c96282356999f7\x00PERSISTENT'       Same time we tried to check "select count(*)" with and without index but it gives difference as shown below -   NOTE: Below output is from test cluster where we were able to repro issue. View name might differ in below screenshot -     We suspected below apache bug for  "WrongRegionException: Requested row out of range for Get on HRegion" -  https://issues.apache.org/jira/browse/PHOENIX-3828   For "select count(*)" mismatch - we suspected we are hitting -  [PHOENIX-6090] Local indexes get out of sync after changes for global consistent indexes - ASF JIRA (apache.org)     Can someone help on debugging steps?   ... View more
Labels:

Re: Ambari UI is responding very Slow

by Explorer in Support Questions
‎01-30-2020 04:14 AM
‎01-30-2020 04:14 AM
@Shelton we are still experiencing slowness in ambari UI after enabling the above properties.    ... View more

Re: Using yarn logs command

by Expert Contributor in Support Questions
‎12-18-2019 10:37 PM
‎12-18-2019 10:37 PM
Hi @Daggers    Please feel free to select best answer if your questions are answered to close the thread.   Thanks ... View more

Re: Different Log directory free space warning on ...

by Community Manager Community Manager in Support Questions
‎12-12-2019 05:42 AM
‎12-12-2019 05:42 AM
@pdev  Can you advise if your issue is now resolved? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.            ... View more

Re: Data replication in datanode new

by Expert Contributor in Support Questions
‎12-03-2019 06:29 AM
‎12-03-2019 06:29 AM
Hi @Peruvian81  there is no such option in ambari UI   You can either check from Namenode UI --> datanode tab and see if the block counts are increasing. ... View more

Re: Ambari server: registration with the server fa...

by Master Mentor in Support Questions
‎12-02-2019 01:27 AM
‎12-02-2019 01:27 AM
@Kou_Bou  Great ! ... View more

Re: Mounting Edge Node Directories to External App...

by Expert Contributor in Support Questions
‎11-29-2019 01:38 AM
‎11-29-2019 01:38 AM
Hi @laplacesdemon    Than you for the response and appreciation. I will be happy to contribute and share my experiences gong further. Thank you for accepting the answer. ... View more

Re: Mysql JDBC connection error for ambari install...

by Contributor in Support Questions
‎11-28-2019 11:05 PM
‎11-28-2019 11:05 PM
It gets error because i already created this user.   mysql> CREATE USER 'ambari'@'%' IDENTIFIED BY 'xxxxxxx'; ERROR 1396 (HY000): Operation CREATE USER failed for 'ambari'@'%' mysql> GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%'; Query OK, 0 rows affected (0.00 sec) mysql> CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'xxxxxx'; ERROR 1396 (HY000): Operation CREATE USER failed for 'ambari'@'localhost' mysql> GRANT ALL PRIVILEGES ON *.* TO 'xxxxxx'@'localhost'; ERROR 1819 (HY000): Your password does not satisfy the current policy requirements mysql> CREATE USER 'ambari'@'xxxxxx' IDENTIFIED BY 'xxxxxx'; ERROR 1396 (HY000): Operation CREATE USER failed for 'ambari'@'xxxxxxx' mysql> GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'xxxxxxx'; Query OK, 0 rows affected (0.00 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) mysql> CREATE USER 'ambari'@'%' IDENTIFIED BY 'xxxxxxxx'; ERROR 1396 (HY000): Operation CREATE USER failed for 'ambari'@'%' mysql> GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%'; Query OK, 0 rows affected (0.00 sec) mysql> mysql> mysql> GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost'; Query OK, 0 rows affected (0.00 sec) mysql> CREATE USER 'ambari'@'xxxxx' IDENTIFIED BY 'xxxx'; ERROR 1396 (HY000): Operation CREATE USER failed for 'ambari'@'xxxxxx' mysql> GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'xxxxxxxx'; Query OK, 0 rows affected (0.00 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.01 sec) ... View more

Re: Random "Clock offset bad" alerts

by Master Guru Master Guru in Support Questions
‎11-28-2019 09:48 AM
‎11-28-2019 09:48 AM
@MihailK does this resolved the issue? If yes, please spare some time to mark this as solution. Thanks. ... View more

Re: Ranger doesn't recognize HDFS and Hive

by New Contributor in Support Questions
‎11-27-2019 03:26 AM
1 Kudo
‎11-27-2019 03:26 AM
1 Kudo
Thanks for the answers. I don't use Kerberos. Adding the services manually to Ranger worked out. It can connect successfully. I don't know why it didn't add the services automatically to Ranger. Anyway my problem is solved. ... View more
Contact Me
Online
Offline
Last Visited
‎10-24-2024 09:03 AM
Community Statistics
Member Since ‎02-18-2016 01:33 AM
Last Visited ‎10-24-2024 09:03 AM
Posts 141
Kudos received 19