Background : Recently my HDP 2.5.6 (dev) cluster got upgraded from HDP 2.3.2.
Since then, I face a weird issue while executing a very simple hive SELECT query.
Before this upgrade, my select query worked well in dev environment.
Fortunately, in production, this SELECT query is working everyday (Production HDP version - 2.3.2). Thank god, production HDP is not yet upgraded :) :)
Hive Version (Dev server):
/etc/hive/conf> hive --version
Hive 1.2.1000.2.5.6.0-40 Subversion git://ctr-e133-1493418528701-94178-01-000004.hwx.site/grid/0/jenkins/workspace/HDP-parallel-suse11sp3/SOURCES/hive -r 4db29b14fe24027bc28690a25058192019075e02
Compiled by jenkins on Mon Jun 26 09:28:06 UTC 2017 From source with checksum 021897025d7879bcef4a8c2f0f8b0949
SELECT query that's giving error follows: (
hive (XYZ_DATABASE)> SELECT count(1) FROM xyz_database.v_fmtrade_big_view v1 inner join
(select max(breakreportingdate) as maxdate FROM xyz_database.crhs_fmtrade_big_table) t1
on v1.breakreportingdate = t1.maxdate;
FAILED: IndexOutOfBoundsException Index: 75, Size: 75
Beauty is, if I split this joined query into two SELECT queries, it works beautifully as follows. But, above joined query is not working after this HDP upgrade. I've checked online if we need to enable/disable any hive parameter. But, couldn't figure out anything.
Anyone faced this kind of error? I'm pretty sure that this needs to attack from parameter setting end - either session level or cluster level.
Query #1
hive (XYZ_DATABASE)> SELECT count(1) FROM XYZ_DATABASE.v_fmtrade_big_view v1;
OK
99453
Time taken: 0.067 seconds, Fetched: 1 row(s)
Query #2:
hive (XYZ_DATABASE)> select max(breakreportingdate) as maxdate FROM XYZ_DATABASE.crhs_fmtrade_big_table;
OK
2017-08-25
Time taken: 12.059 seconds, Fetched: 1 row(s)
Below Error Stacktrace is from /tmp/<userid>/hive.log file
(SessionState.java:printError(989)) - FAILED: IndexOutOfBoundsException Index: 75, Size: 75 java.lang.IndexOutOfBoundsException: Index: 75, Size: 75 at java.util.ArrayList.rangeCheck(ArrayList.java:653) at java.util.ArrayList.get(ArrayList.java:429) at org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerSelectProc.process(ColumnPrunerProcFactory.java:782) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79) at org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:176) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110) at org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:136) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:207) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10464) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:218) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:230) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:464) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:320) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1219) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1260) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:217) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:169) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:380) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:740) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:685) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
