Hi all, I've been looking for solutions to expose Kafka data in Hive. I found a few things, however HiveKafkaStorageHandler caught my attention :). I was looking for trying it out, but I don't find anywhere in the documentation how does it support a connection to Kafka through SSL (which is a requirement for my setup). Can someone tell me for sure if this storage handler supports a broker listener that uses SSL? Thanks a lot in advance ! ... View more

Hi all, I am getting an ArrayIndexOutOfBoundsException on a phoenix query and I would like to know if you have any suggestion to solve this problem. The error is throwing when I select data from some tables to upsert into another. The select query gets all the data from a table that has a considerable amount of data (3 374 590 registers) and merges with data from 5 smaller tables. I have 4 DataNode/RegiongServer/PhoenixQS (almost dedicated) nodes in the cluster (6 vCPU; 32GB RAM per node), so I believe that resources are not the problem. As a workarround I am filtering by date in order to be able to transfer the data between the tables. It's odd to notice that sometimes I am able to upsert 50k register in a query (a few days), and sometimes I am limited to 9k registers (arround 2 days) or less. I get the error even using a hint to change the join algorithm. An exemple of the errors I am getting: Error: java.lang.ArrayIndexOutOfBoundsException: -6 (state=08000,code=101) org.apache.phoenix.exception.PhoenixIOException: java.lang.ArrayIndexOutOfBoundsException: -6 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:538) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:510) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:737) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:305) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:297) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:295) at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1255) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:808) at sqlline.SqlLine.begin(SqlLine.java:681) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:292) Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: -6 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:202) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:534) ... 16 more Caused by: java.lang.ArrayIndexOutOfBoundsException: -6 at org.apache.phoenix.util.ByteUtil.vlongFromBytes(ByteUtil.java:329) at org.apache.phoenix.util.ByteUtil.vintFromBytes(ByteUtil.java:316) at org.apache.phoenix.schema.KeyValueSchema.next(KeyValueSchema.java:208) at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:165) at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:171) at org.apache.phoenix.schema.KeyValueSchema.iterator(KeyValueSchema.java:175) at org.apache.phoenix.expression.ProjectedColumnExpression.evaluate(ProjectedColumnExpression.java:112) at org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69) at org.apache.phoenix.jdbc.PhoenixResultSet.getObject(PhoenixResultSet.java:515) at org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:164) at org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:105) at org.apache.phoenix.compile.UpsertCompiler$UpsertingParallelIteratorFactory.mutate(UpsertCompiler.java:221) at org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(MutatingParallelIteratorFactory.java:61) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:99) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:90) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:172) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745 I am using HDP 2.4.0, so phoenix 4.4. The configurations I am using are: HBase RegionServer Maximum Memory=12288 (12GB) HBase Master Maximum Memory=12288 (12GB) Number of Handlers per RegionServer=30 Memstore Flush Size=128MB Maximum Record Size=1MB Maximum Region File Size=10GB % of RegionServer Allocated to Read Buffers=40% % of RegionServer Allocated to Write Buffers=40% HBase RPC Timeout=6min Zookeeper Session Timeout=6min Phoenix Query Timeout=6min Number of Fetched Rows when Scanning from Disk=10000 dfs.client.read.shortcircuit=true dfs.client.read.shortcircuit.buffer.size=131072 hbase.hstore.min.locality.to.skip.major.compact=0.7 hbase.ipc.server.callqueue.read.ratio=0.8 hbase.ipc.server.callqueue.scan.ratio=0.8 phoenix.coprocessor.maxServerCacheTimeToLiveMs=30000 phoenix.mutate.batchSize=100000 phoenix.query.maxServerCacheBytes=8589934592 phoenix.query.queueSize=7500 phoenix.query.threadPoolSize=512 The other configurations are the default. What should I change in order to make a query of this size work properly? Thanks in advance. ... View more