Can you disable Column Pruner?

wgonzalez — Thu, 08 Oct 2015 02:18:53 GMT

Hive v13,hitting the semantic error:

2015-10-07 05:12:40,988 ERROR ql.Driver (SessionState.java:printError(547)) - FAILED: NullPointerException null java.lang.NullPointerException at org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerSelectProc.process(ColumnPrunerProcFactory.java:618) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) at org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:166) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) at org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:129) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9414) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:359) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:456) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:466) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:749) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Looks very similar to:

https://issues.apache.org/jira/browse/HIVE-9185

Looks like the ability to disable pruner was taken away on Hive13:

hive.optimize.cp=false

https://issues.apache.org/jira/browse/HIVE-4113

Is there another way to disable the pruner temporarily?

Re: Can you disable Column Pruner?

deepesh1 — Thu, 08 Oct 2015 06:30:16 GMT

From the code it looks like ColumnPruner is always applied by optimizer and there is no way to exclude it after HIVE-4113. It would be good to get more details on the scenario where you ran into this. Have you tried this query with Hive v0.14 or later?

Re: Can you disable Column Pruner?

wgonzalez — Thu, 08 Oct 2015 22:37:56 GMT

Yes and it gave more insight into what was wrong:

FAILED: SemanticException [Error 10007]: Ambiguous column reference visit_num in features_and_label

Customer had a duped column.

Thank you for the tip.

question Can you disable Column Pruner? in Archives of Support Questions (Read Only)

Can you disable Column Pruner?

Re: Can you disable Column Pruner?

Re: Can you disable Column Pruner?