Phoenix query throwing ArrayIndexOutOfBoundsException

[#1]

Hi all,

I am getting an ArrayIndexOutOfBoundsException on a phoenix query and I would like to know if you have any suggestion to solve this problem.

The error is throwing when I select data from some tables to upsert into another. The select query gets all the data from a table that has a considerable amount of data (3 374 590 registers) and merges with data from 5 smaller tables.

I have 4 DataNode/RegiongServer/PhoenixQS (almost dedicated) nodes in the cluster (6 vCPU; 32GB RAM per node), so I believe that resources are not the problem. As a workarround I am filtering by date in order to be able to transfer the data between the tables. It's odd to notice that sometimes I am able to upsert 50k register in a query (a few days), and sometimes I am limited to 9k registers (arround 2 days) or less. I get the error even using a hint to change the join algorithm.

An exemple of the errors I am getting:

Error: java.lang.ArrayIndexOutOfBoundsException: -6 (state=08000,code=101)
org.apache.phoenix.exception.PhoenixIOException: java.lang.ArrayIndexOutOfBoundsException: -6
	at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:538)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:510)
	at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
	at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
	at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
	at org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:737)
	at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:305)
	at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:297)
	at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:295)
	at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1255)
	at sqlline.Commands.execute(Commands.java:822)
	at sqlline.Commands.sql(Commands.java:732)
	at sqlline.SqlLine.dispatch(SqlLine.java:808)
	at sqlline.SqlLine.begin(SqlLine.java:681)
	at sqlline.SqlLine.start(SqlLine.java:398)
	at sqlline.SqlLine.main(SqlLine.java:292)
Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: -6
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:202)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:534)
	... 16 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -6
	at org.apache.phoenix.util.ByteUtil.vlongFromBytes(ByteUtil.java:329)
	at org.apache.phoenix.util.ByteUtil.vintFromBytes(ByteUtil.java:316)
	at org.apache.phoenix.schema.KeyValueSchema.next(KeyValueSchema.java:208)
	at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:165)
	at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:171)
	at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:175)
	at org.apache.phoenix.expression.ProjectedColumnExpression.evaluate(ProjectedColumnExpression.java:112)
	at org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69)
	at org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:515)
	at org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:164)
	at org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:105)
	at org.apache.phoenix.compile.UpsertCompiler$UpsertingParallelIteratorFactory.mutate(UpsertCompiler.java:221)
	at org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(MutatingParallelIteratorFactory.java:61)
	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:99)
	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:90)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:172)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745

I am using HDP 2.4.0, so phoenix 4.4. The configurations I am using are:

HBase RegionServer Maximum Memory=12288 (12GB)
HBase Master Maximum Memory=12288 (12GB)
Number of Handlers per RegionServer=30
Memstore Flush Size=128MB
Maximum Record Size=1MB
Maximum Region File Size=10GB
% of RegionServer Allocated to Read Buffers=40%
% of RegionServer Allocated to Write Buffers=40%

HBase RPC Timeout=6min
Zookeeper Session Timeout=6min
Phoenix Query Timeout=6min

Number of Fetched Rows when Scanning from Disk=10000
dfs.client.read.shortcircuit=true
dfs.client.read.shortcircuit.buffer.size=131072
hbase.hstore.min.locality.to.skip.major.compact=0.7
hbase.ipc.server.callqueue.read.ratio=0.8
hbase.ipc.server.callqueue.scan.ratio=0.8
phoenix.coprocessor.maxServerCacheTimeToLiveMs=30000
phoenix.mutate.batchSize=100000
phoenix.query.maxServerCacheBytes=8589934592
phoenix.query.queueSize=7500
phoenix.query.threadPoolSize=512

The other configurations are the default.

What should I change in order to make a query of this size work properly?

Thanks in advance.

[#2]

Can you please share the version of HDP you're running, the table's schema and the UPSERT SELECT command you ran?

[#3]

Hi Josh, I am using HDP 2.4.0. The query I am using looks like the following:

upsert into TABLE_HBASE_DENORM 
   select 
    TABLE_A.MPK,
    TABLE_A.AAAA,
    TABLE_A.BBBB,
    TABLE_A.CCCC,
    TABLE_A.DDDD,
    TABLE_A.EEEE,
    TABLE_B.AAA,
    TABLE_B.BBB,
    TABLE_B.CCC,
... 26 other parameters ...
    TABLE_A.M4,
    TABLE_A.M3,
    TABLE_A.M2,
    TABLE_A.M1,
    TABLE_A.Q4,
    TABLE_A.Q3,
    TABLE_A.Q2,
    TABLE_A.Q1
   from  
    TABLE_A,
    TABLE_B,
    TABLE_C,
    TABLE_F,
    TABLE_E,
    TABLE_D
   where 
    TABLE_A.AAAA = TABLE_B.AAAA AND
    TABLE_A.BBBB = TABLE_C.BBBB AND
    TABLE_A.CCCC = TABLE_D.CCCC AND   
    TABLE_A.DDDD = TABLE_E.DDDD AND
    TABLE_A.EEEE = TABLE_F.EEEE AND
    
    TABLE_A.Q3 >= TO_TIMESTAMP('2017-03-04 10:40:05') AND TABLE_A.Q3 < TO_TIMESTAMP('2017-03-05 10:40:05') 
 ;

[#4 · Accepted solution]

It may be due to https://issues.apache.org/jira/browse/PHOENIX-2169

bq. It's odd to notice that sometimes I am able to upsert 50k register in a query (a few days), and sometimes I am limited to 9k registers (around 2 days) or less

It may happen irregularly when UPSERT SELECT running scans and mutating parallely as our ProjectedColumnExpression is not thread safe. so you may try backporting PHOENIX-2169 in your distribution or upgrade to HDP 2.5 or PHOENIX-4.7.

[#5]

Just updating phoenix to version 4.7 would solve this issue then? Is it possible to update just the Phoenix version wihtout updating the whole cluster?

Thank you.

[#6]

I'll not recommend updating a single component in a stack, as there will be some incompatibility and upgrade issues.

I only suspect you might be hitting PHOENIX-2169 so it's better to reproduce the same issue somewhere in pre-prod/dev and see if the fix works and then replicate on production.

Or, you can talk to your vendor to provide the hotfix for the same version.

[#7]

I will try to reproduce the situation and then apply a fix!

Thanks a lot for your feedback.