Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎02-05-2014

Phoenix upgrade failure from CDH5.4.2 to CDH5.5.2

We upgraded our PROD cluster from CDH5.4.2 to 5.5.2 and also upgraded phoenix to CDH5.5.2 compatible. After upgrade Phoenix queries are failing, Hbase is working fine. We were able to upgrade our lower enviornment without any issue.

 

Can some one help me to find the root cause ?

 

Here's the error message we are seeing in sqlline.py console.

 

16/03/26 09:27:09 INFO client.RpcRetryingCaller: Call exception, tries=11, retries=35, started=448408 ms ago, cancelled=false, msg=row 'SYSTEMCATALOG' on table 'SYSTEM.CATALOG' at region=SYSTEM.CATALOG,,1456593238403.04dabf57a618035bf7f939e4fcecf945., hostname=hadoopclusterp008.prod.con,60020,1458997191463, seqNum=165

 

Here's is the Region server log message

 

2016-03-26 10:02:11,771 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=46, retries=350, started=792139 ms ago, cancelled=false, msg=row 'SYSTEM.CATALOG' on table 'SYSTEM.STATS' at region=SYSTEM.STATS,,1456593918087.ed3fe1b38be6fe2ad4b198de63e842a8., hostname=hadoopclusterp008.prod.con,60020,1456593833481, seqNum=2
2016-03-26 10:02:31,826 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=47, retries=350, started=812195 ms ago, cancelled=false, msg=row 'SYSTEM.CATALOG' on table 'SYSTEM.STATS' at region=SYSTEM.STATS,,1456593918087.ed3fe1b38be6fe2ad4b198de63e842a8., hostname=hadoopclusterp008.prod.con,60020,1456593833481, seqNum=2
2016-03-26 10:02:34,737 ERROR org.apache.phoenix.coprocessor.MetaDataEndpointImpl: createTable failed
java.io.IOException: Timed out waiting for lock for row: \x00SYSTEM\x00CATALOG
at org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:4824)
at org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:4793)
at org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:4846)
at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.acquireLock(MetaDataEndpointImpl.java:1273)
at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.createTable(MetaDataEndpointImpl.java:1161)
at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:11619)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7388)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1776)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1758)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32209)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)

 

 

Cloudera Employee
Posts: 7
Registered: ‎09-18-2013

Re: Phoenix upgrade failure from CDH5.4.2 to CDH5.5.2

Looks related to this 

 

https://issues.apache.org/jira/browse/PHOENIX-2508

 

Is this something you can reproduce reliably? (even after restarts, etc).

 

Thanks

Jon

Jonathan Hsieh (shay)
Software Engineer, Cloudera
Cloudera Employee
Posts: 7
Registered: ‎09-18-2013

Re: Phoenix upgrade failure from CDH5.4.2 to CDH5.5.2

Also from this related thread[1], you might be able to get out of this state by dropping the SYSTEM.CATALOG table and restarting.

YMMV -- I personally haven't tried this, so to be safe you may want to snapshot backup SYSTEM.CATALOG before dropping!

[1]https://groups.google.com/forum/#!topic/phoenix-hbase-user/c7Yt8Dedf1s
Jonathan Hsieh (shay)
Software Engineer, Cloudera
New Contributor
Posts: 2
Registered: ‎02-05-2014

Re: Phoenix upgrade failure from CDH5.4.2 to CDH5.5.2

Hi Jon,

 

 Thanks for the reply.  I had to delete all 4 system tables to bring Phoenix work.

 

Now I am unable to see any phoenix tables in Phoenix, I can see all tables in Hbase. Is there a way to refresh tables inorder to add older phoenix tables (which exists in Hbase) to Phoenix metadata ?

 

Or Should I recreate this table, Our Production upgrade is is planned this week, so we need to find another alternate solution, Recreating all tables in Production is a tedious task, as we have lots of table in PROD.

 

Thanks,

Sachin

 

 

 

 

 

 

 

Cloudera Employee
Posts: 7
Registered: ‎09-18-2013

Re: Phoenix upgrade failure from CDH5.4.2 to CDH5.5.2

The suggestion was to backup and then only drop the SYSTEM.CATALOG table and no other system tables. 

 

You are going into territory that I beyond my area of expertise.

 

Can you restore the other system tables?  Can you answer the rest of the questions?

 

Thanks

Jon.

Jonathan Hsieh (shay)
Software Engineer, Cloudera