Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Phoenix query throwing ArrayIndexOutOfBoundsException

avatar

Hi all,

I am getting an ArrayIndexOutOfBoundsException on a phoenix query and I would like to know if you have any suggestion to solve this problem.

The error is throwing when I select data from some tables to upsert into another. The select query gets all the data from a table that has a considerable amount of data (3 374 590 registers) and merges with data from 5 smaller tables.

I have 4 DataNode/RegiongServer/PhoenixQS (almost dedicated) nodes in the cluster (6 vCPU; 32GB RAM per node), so I believe that resources are not the problem. As a workarround I am filtering by date in order to be able to transfer the data between the tables. It's odd to notice that sometimes I am able to upsert 50k register in a query (a few days), and sometimes I am limited to 9k registers (arround 2 days) or less. I get the error even using a hint to change the join algorithm.

An exemple of the errors I am getting:

Error: java.lang.ArrayIndexOutOfBoundsException: -6 (state=08000,code=101)
org.apache.phoenix.exception.PhoenixIOException: java.lang.ArrayIndexOutOfBoundsException: -6
	at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:538)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:510)
	at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
	at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
	at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
	at org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:737)
	at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:305)
	at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:297)
	at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:295)
	at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1255)
	at sqlline.Commands.execute(Commands.java:822)
	at sqlline.Commands.sql(Commands.java:732)
	at sqlline.SqlLine.dispatch(SqlLine.java:808)
	at sqlline.SqlLine.begin(SqlLine.java:681)
	at sqlline.SqlLine.start(SqlLine.java:398)
	at sqlline.SqlLine.main(SqlLine.java:292)
Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: -6
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:202)
	at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:534)
	... 16 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -6
	at org.apache.phoenix.util.ByteUtil.vlongFromBytes(ByteUtil.java:329)
	at org.apache.phoenix.util.ByteUtil.vintFromBytes(ByteUtil.java:316)
	at org.apache.phoenix.schema.KeyValueSchema.next(KeyValueSchema.java:208)
	at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:165)
	at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:171)
	at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:175)
	at org.apache.phoenix.expression.ProjectedColumnExpression.evaluate(ProjectedColumnExpression.java:112)
	at org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69)
	at org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:515)
	at org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:164)
	at org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:105)
	at org.apache.phoenix.compile.UpsertCompiler$UpsertingParallelIteratorFactory.mutate(UpsertCompiler.java:221)
	at org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(MutatingParallelIteratorFactory.java:61)
	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:99)
	at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:90)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:172)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745

I am using HDP 2.4.0, so phoenix 4.4. The configurations I am using are:

HBase RegionServer Maximum Memory=12288 (12GB)
HBase Master Maximum Memory=12288 (12GB)
Number of Handlers per RegionServer=30
Memstore Flush Size=128MB
Maximum Record Size=1MB
Maximum Region File Size=10GB
% of RegionServer Allocated to Read Buffers=40%
% of RegionServer Allocated to Write Buffers=40%

HBase RPC Timeout=6min
Zookeeper Session Timeout=6min
Phoenix Query Timeout=6min

Number of Fetched Rows when Scanning from Disk=10000
dfs.client.read.shortcircuit=true
dfs.client.read.shortcircuit.buffer.size=131072
hbase.hstore.min.locality.to.skip.major.compact=0.7
hbase.ipc.server.callqueue.read.ratio=0.8
hbase.ipc.server.callqueue.scan.ratio=0.8
phoenix.coprocessor.maxServerCacheTimeToLiveMs=30000
phoenix.mutate.batchSize=100000
phoenix.query.maxServerCacheBytes=8589934592
phoenix.query.queueSize=7500
phoenix.query.threadPoolSize=512

The other configurations are the default.

What should I change in order to make a query of this size work properly?

Thanks in advance.

1 ACCEPTED SOLUTION

avatar

It may be due to https://issues.apache.org/jira/browse/PHOENIX-2169

bq. It's odd to notice that sometimes I am able to upsert 50k register in a query (a few days), and sometimes I am limited to 9k registers (around 2 days) or less

It may happen irregularly when UPSERT SELECT running scans and mutating parallely as our ProjectedColumnExpression is not thread safe. so you may try backporting PHOENIX-2169 in your distribution or upgrade to HDP 2.5 or PHOENIX-4.7.

View solution in original post

6 REPLIES 6

avatar
Super Guru

Can you please share the version of HDP you're running, the table's schema and the UPSERT SELECT command you ran?

avatar

Hi Josh, I am using HDP 2.4.0. The query I am using looks like the following:

upsert into TABLE_HBASE_DENORM 
   select 
    TABLE_A.MPK,
    TABLE_A.AAAA,
    TABLE_A.BBBB,
    TABLE_A.CCCC,
    TABLE_A.DDDD,
    TABLE_A.EEEE,
    TABLE_B.AAA,
    TABLE_B.BBB,
    TABLE_B.CCC,
... 26 other parameters ...
    TABLE_A.M4,
    TABLE_A.M3,
    TABLE_A.M2,
    TABLE_A.M1,
    TABLE_A.Q4,
    TABLE_A.Q3,
    TABLE_A.Q2,
    TABLE_A.Q1
   from  
    TABLE_A,
    TABLE_B,
    TABLE_C,
    TABLE_F,
    TABLE_E,
    TABLE_D
   where 
    TABLE_A.AAAA = TABLE_B.AAAA AND
    TABLE_A.BBBB = TABLE_C.BBBB AND
    TABLE_A.CCCC = TABLE_D.CCCC AND   
    TABLE_A.DDDD = TABLE_E.DDDD AND
    TABLE_A.EEEE = TABLE_F.EEEE AND
    
    TABLE_A.Q3 >= TO_TIMESTAMP('2017-03-04 10:40:05') AND TABLE_A.Q3 < TO_TIMESTAMP('2017-03-05 10:40:05') 
 ;

avatar

It may be due to https://issues.apache.org/jira/browse/PHOENIX-2169

bq. It's odd to notice that sometimes I am able to upsert 50k register in a query (a few days), and sometimes I am limited to 9k registers (around 2 days) or less

It may happen irregularly when UPSERT SELECT running scans and mutating parallely as our ProjectedColumnExpression is not thread safe. so you may try backporting PHOENIX-2169 in your distribution or upgrade to HDP 2.5 or PHOENIX-4.7.

avatar

Just updating phoenix to version 4.7 would solve this issue then? Is it possible to update just the Phoenix version wihtout updating the whole cluster?

Thank you.

avatar

I'll not recommend updating a single component in a stack, as there will be some incompatibility and upgrade issues.

I only suspect you might be hitting PHOENIX-2169 so it's better to reproduce the same issue somewhere in pre-prod/dev and see if the fix works and then replicate on production.

Or, you can talk to your vendor to provide the hotfix for the same version.

avatar

I will try to reproduce the situation and then apply a fix!

Thanks a lot for your feedback.