- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Phoenix query throwing ArrayIndexOutOfBoundsException
- Labels:
-
Apache HBase
-
Apache Phoenix
Created ‎07-22-2017 08:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi all,
I am getting an ArrayIndexOutOfBoundsException on a phoenix query and I would like to know if you have any suggestion to solve this problem.
The error is throwing when I select data from some tables to upsert into another. The select query gets all the data from a table that has a considerable amount of data (3 374 590 registers) and merges with data from 5 smaller tables.
I have 4 DataNode/RegiongServer/PhoenixQS (almost dedicated) nodes in the cluster (6 vCPU; 32GB RAM per node), so I believe that resources are not the problem. As a workarround I am filtering by date in order to be able to transfer the data between the tables. It's odd to notice that sometimes I am able to upsert 50k register in a query (a few days), and sometimes I am limited to 9k registers (arround 2 days) or less. I get the error even using a hint to change the join algorithm.
An exemple of the errors I am getting:
Error: java.lang.ArrayIndexOutOfBoundsException: -6 (state=08000,code=101) org.apache.phoenix.exception.PhoenixIOException: java.lang.ArrayIndexOutOfBoundsException: -6 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:538) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:510) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:737) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:305) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:297) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:295) at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1255) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:808) at sqlline.SqlLine.begin(SqlLine.java:681) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:292) Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: -6 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:202) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:534) ... 16 more Caused by: java.lang.ArrayIndexOutOfBoundsException: -6 at org.apache.phoenix.util.ByteUtil.vlongFromBytes(ByteUtil.java:329) at org.apache.phoenix.util.ByteUtil.vintFromBytes(ByteUtil.java:316) at org.apache.phoenix.schema.KeyValueSchema.next(KeyValueSchema.java:208) at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:165) at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:171) at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:175) at org.apache.phoenix.expression.ProjectedColumnExpression.evaluate(ProjectedColumnExpression.java:112) at org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69) at org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:515) at org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:164) at org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:105) at org.apache.phoenix.compile.UpsertCompiler$UpsertingParallelIteratorFactory.mutate(UpsertCompiler.java:221) at org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(MutatingParallelIteratorFactory.java:61) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:99) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:90) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:172) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745
I am using HDP 2.4.0, so phoenix 4.4. The configurations I am using are:
HBase RegionServer Maximum Memory=12288 (12GB) HBase Master Maximum Memory=12288 (12GB) Number of Handlers per RegionServer=30 Memstore Flush Size=128MB Maximum Record Size=1MB Maximum Region File Size=10GB % of RegionServer Allocated to Read Buffers=40% % of RegionServer Allocated to Write Buffers=40% HBase RPC Timeout=6min Zookeeper Session Timeout=6min Phoenix Query Timeout=6min Number of Fetched Rows when Scanning from Disk=10000 dfs.client.read.shortcircuit=true dfs.client.read.shortcircuit.buffer.size=131072 hbase.hstore.min.locality.to.skip.major.compact=0.7 hbase.ipc.server.callqueue.read.ratio=0.8 hbase.ipc.server.callqueue.scan.ratio=0.8 phoenix.coprocessor.maxServerCacheTimeToLiveMs=30000 phoenix.mutate.batchSize=100000 phoenix.query.maxServerCacheBytes=8589934592 phoenix.query.queueSize=7500 phoenix.query.threadPoolSize=512
The other configurations are the default.
What should I change in order to make a query of this size work properly?
Thanks in advance.
Created ‎07-26-2017 06:06 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It may be due to https://issues.apache.org/jira/browse/PHOENIX-2169
bq. It's odd to notice that sometimes I am able to upsert 50k register in a query (a few days), and sometimes I am limited to 9k registers (around 2 days) or less
It may happen irregularly when UPSERT SELECT running scans and mutating parallely as our ProjectedColumnExpression is not thread safe. so you may try backporting PHOENIX-2169 in your distribution or upgrade to HDP 2.5 or PHOENIX-4.7.
Created ‎07-25-2017 04:01 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you please share the version of HDP you're running, the table's schema and the UPSERT SELECT command you ran?
Created ‎07-27-2017 09:12 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Josh, I am using HDP 2.4.0. The query I am using looks like the following:
upsert into TABLE_HBASE_DENORM select TABLE_A.MPK, TABLE_A.AAAA, TABLE_A.BBBB, TABLE_A.CCCC, TABLE_A.DDDD, TABLE_A.EEEE, TABLE_B.AAA, TABLE_B.BBB, TABLE_B.CCC, ... 26 other parameters ... TABLE_A.M4, TABLE_A.M3, TABLE_A.M2, TABLE_A.M1, TABLE_A.Q4, TABLE_A.Q3, TABLE_A.Q2, TABLE_A.Q1 from TABLE_A, TABLE_B, TABLE_C, TABLE_F, TABLE_E, TABLE_D where TABLE_A.AAAA = TABLE_B.AAAA AND TABLE_A.BBBB = TABLE_C.BBBB AND TABLE_A.CCCC = TABLE_D.CCCC AND TABLE_A.DDDD = TABLE_E.DDDD AND TABLE_A.EEEE = TABLE_F.EEEE AND TABLE_A.Q3 >= TO_TIMESTAMP('2017-03-04 10:40:05') AND TABLE_A.Q3 < TO_TIMESTAMP('2017-03-05 10:40:05') ;
Created ‎07-26-2017 06:06 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It may be due to https://issues.apache.org/jira/browse/PHOENIX-2169
bq. It's odd to notice that sometimes I am able to upsert 50k register in a query (a few days), and sometimes I am limited to 9k registers (around 2 days) or less
It may happen irregularly when UPSERT SELECT running scans and mutating parallely as our ProjectedColumnExpression is not thread safe. so you may try backporting PHOENIX-2169 in your distribution or upgrade to HDP 2.5 or PHOENIX-4.7.
Created ‎07-27-2017 09:15 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just updating phoenix to version 4.7 would solve this issue then? Is it possible to update just the Phoenix version wihtout updating the whole cluster?
Thank you.
Created ‎07-27-2017 10:11 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'll not recommend updating a single component in a stack, as there will be some incompatibility and upgrade issues.
I only suspect you might be hitting PHOENIX-2169 so it's better to reproduce the same issue somewhere in pre-prod/dev and see if the fix works and then replicate on production.
Or, you can talk to your vendor to provide the hotfix for the same version.
Created ‎07-28-2017 10:32 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I will try to reproduce the situation and then apply a fix!
Thanks a lot for your feedback.
