Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Phoenix 5.0 startup error using sqlline.py with HDP 3.1.5 (HBase 2.1.6.3.1.5.0-152)

avatar
New Contributor

Getting the below error, when I start the Phoenix5.0 using sqlline.py first time after install HDP 3.1.5 version.

 

 

Error: org.apache.hadoop.hbase.DoNotRetryIOException: SYSTEM.CATALOG: Didn't find expected key values for table row in metadata row
	at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:114)
	at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:596)
	at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:16357)
	at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8243)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2444)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2426)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42198)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.IllegalStateException: Didn't find expected key values for table row in metadata row
	at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:918)
	at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:612)
	at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3448)
	at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3392)
	at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:554)
	... 9 more (state=08000,code=101)
org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: SYSTEM.CATALOG: Didn't find expected key values for table row in metadata row

 

 

From Hbase I don't see any issue as I am able to see table list and table row count etc..
Can anyone help how to fix this? 

3 REPLIES 3

avatar
Explorer

1. shutdown hbase master and regionserver ;
2. execute 'hbase clean --cleanZk'

avatar
New Contributor

Thanks..It might work for the fresh installation of HDP 3.1.5. Actually I have restored data from the HDP 2.6.2 cluster and then I am facing this issue. 
For the solution I have truncated the SYSTEM.CATALOG metadata table and restored data again from the old cluster, then its working for me with out any issue.

But I am facing the issue, when trying to connect using Aqua studio by phoenix 5.0 client.jar, getting the below error.

java.sql.SQLException: ERROR 1102 (XCL02): Cannot get all table regions
....................
Caused by: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: Is this a pre-hbase-1.0.0 or asynchbase client? Client is invoking getClosestRowBefore removed in hbase-2.0.0 replaced by reverse Scan 

Can anyone please help me how to resolve this issue?

avatar
Explorer

@Karee I encountered this error message when I run sqlline.py of phoenix 4.7 against Hbase 2.0 (which is using phoenix 5.0), so please make sure you're really using phoenix 5.0 client jar.

 

In addition, I'm upgrading Hbase from 1.2 to 2.0, then I got the same issue as your (SYSTEM.CATALOG: Didn't find expected key values for table row in metadata row) when I ran sqlline.py, do you have a idea what's the real root cause? Thanks!