Created 09-18-2020 01:52 AM
Getting the below error, when I start the Phoenix5.0 using sqlline.py first time after install HDP 3.1.5 version.
Error: org.apache.hadoop.hbase.DoNotRetryIOException: SYSTEM.CATALOG: Didn't find expected key values for table row in metadata row
at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:114)
at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:596)
at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:16357)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8243)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2444)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2426)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42198)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.IllegalStateException: Didn't find expected key values for table row in metadata row
at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:918)
at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:612)
at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3448)
at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3392)
at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:554)
... 9 more (state=08000,code=101)
org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: SYSTEM.CATALOG: Didn't find expected key values for table row in metadata row
From Hbase I don't see any issue as I am able to see table list and table row count etc..
Can anyone help how to fix this?
Created 09-19-2020 11:28 PM
1. shutdown hbase master and regionserver ;
2. execute 'hbase clean --cleanZk'
Created 09-23-2020 01:19 AM
Thanks..It might work for the fresh installation of HDP 3.1.5. Actually I have restored data from the HDP 2.6.2 cluster and then I am facing this issue.
For the solution I have truncated the SYSTEM.CATALOG metadata table and restored data again from the old cluster, then its working for me with out any issue.
But I am facing the issue, when trying to connect using Aqua studio by phoenix 5.0 client.jar, getting the below error.
java.sql.SQLException: ERROR 1102 (XCL02): Cannot get all table regions
....................
Caused by: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: Is this a pre-hbase-1.0.0 or asynchbase client? Client is invoking getClosestRowBefore removed in hbase-2.0.0 replaced by reverse Scan
Can anyone please help me how to resolve this issue?
Created 10-12-2020 07:35 AM
@Karee I encountered this error message when I run sqlline.py of phoenix 4.7 against Hbase 2.0 (which is using phoenix 5.0), so please make sure you're really using phoenix 5.0 client jar.
In addition, I'm upgrading Hbase from 1.2 to 2.0, then I got the same issue as your (SYSTEM.CATALOG: Didn't find expected key values for table row in metadata row) when I ran sqlline.py, do you have a idea what's the real root cause? Thanks!