Apache Phoenix - ERROR 1108 (XCL08): Cache of region boundaries are out of date.

[#1]

Hello !

We're using Apace Phoenix 4.13.2 with CDH 5.13.0.

When I'm trying to get count on some big table (~1,4 billion records) I'm getting

Remote driver error: RuntimeException: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are o
ut of date. -> StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date.
        at org.apache.phoenix.shaded.org.apache.calcite.avatica.Helper.createException(Helper.java:54)
        at org.apache.phoenix.shaded.org.apache.calcite.avatica.Helper.createException(Helper.java:41)
        at org.apache.phoenix.shaded.org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
        at org.apache.phoenix.shaded.org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:209)
        at sqlline.Commands.execute(Commands.java:822)
        at sqlline.Commands.sql(Commands.java:732)
        at sqlline.SqlLine.dispatch(SqlLine.java:813)
        at sqlline.SqlLine.begin(SqlLine.java:686)
        at sqlline.SqlLine.start(SqlLine.java:398)
        at sqlline.SqlLine.main(SqlLine.java:291)
        at org.apache.phoenix.queryserver.client.SqllineWrapper.main(SqllineWrapper.java:93)
java.lang.RuntimeException: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date.
        at org.apache.calcite.avatica.jdbc.JdbcResultSet.create(JdbcResultSet.java:109)
        at org.apache.calcite.avatica.jdbc.JdbcResultSet.create(JdbcResultSet.java:81)
        at org.apache.calcite.avatica.jdbc.JdbcMeta.prepareAndExecute(JdbcMeta.java:742)
        at org.apache.calcite.avatica.remote.LocalService.apply(LocalService.java:206)
        at org.apache.calcite.avatica.remote.Service$PrepareAndExecuteRequest.accept(Service.java:928)
        at org.apache.calcite.avatica.remote.Service$PrepareAndExecuteRequest.accept(Service.java:880)
        at org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:94)
        at org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46)
        at org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:127)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
        at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date.
        at org.apache.phoenix.exception.SQLExceptionCode$14.newException(SQLExceptionCode.java:361)
        at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
        at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:146)
        at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:130)
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:116)
        at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:127)
        at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:166)
        at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
        at org.apache.phoenix.iterate.LookAheadResultIterator.next(LookAheadResultIterator.java:67)
        at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
        at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
        at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:79)
        at org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
        at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:779)
        at org.apache.calcite.avatica.jdbc.JdbcResultSet.frame(JdbcResultSet.java:148)
        at org.apache.calcite.avatica.jdbc.JdbcResultSet.create(JdbcResultSet.java:101)
        ... 17 more

Tried to do rolling restart for HBase - doesn't help;

Tried to delete rows from system.stats - doesn't help.

Can someone help ?

[#2 · Accepted solution]

That was caused by different values of 

hbase.client.scanner.timeout.period

on Apache Phoenix QueryServer and HBase. After set that to the same value - works like a charm.