Out of Memory Error in Hive

[#1]

I am getting the below error while trying to execute a query like "select * from a where a.col1 not in (select b.col1 from b)"

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

at java.util.Arrays.copyOfRange(Arrays.java:2694)

at java.lang.String.<init>(String.java:203)

at java.lang.StringBuilder.toString(StringBuilder.java:405)

at org.apache.hadoop.fs.Path.toString(Path.java:390)

at org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.getBucketFilePathsOfPartition(AbstractBucketJoinProc.java:87)

at org.apache.hadoop.hive.ql.optimizer.metainfo.annotation.OpTraitsRulesProcFactory$TableScanRule.checkBucketedTable(OpTraitsRulesProcFactory.java:147)

at org.apache.hadoop.hive.ql.optimizer.metainfo.annotation.OpTraitsRulesProcFactory$TableScanRule.process(OpTraitsRulesProcFactory.java:174)

at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)

at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:56)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)

at org.apache.hadoop.hive.ql.optimizer.metainfo.annotation.AnnotateWithOpTraits.transform(AnnotateWithOpTraits.java:91)

at org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsAnnotation(TezCompiler.java:249)

at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:122)

at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:102)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10188)

at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:211)

at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)

at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)

at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)

at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)

at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)

at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)

at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

I tried increasing value for below properties, but it is not working. Hive is running on TEZ

mapreduce.map.memory.mb

mapreduce.reduce.memory.mb

hive.tez.container.size

hive.tez.java.opts

[#2]

You might want to take a look at this

https://community.hortonworks.com/questions/35403/hive-query-outofmemoryerror-java-heap-space.html

[#3]

Hello pooja

From your stack trace your table seems to be bucketed. Can you share your table definition

could you also try running the query with the setting: hive.auto.convert.join.noconditionaltask=false

[#4]

Hello,

I had tried with hive.auto.convert.join.noconditionaltask=false, but didn't work.

No table is bucketed.

[#5]

@nmaillard

I am have a same problem. I am on http://host:8080/#/main/services/HIVE/configs but not sure what to change in the configs.

[#6 · Accepted solution]

The problem is probably because there is too much data moving through the shuffle phase. You can reduce the amount of data moving between tasks as part of the SHUFFLE steps by using more aggressive queries and by looking carefully at your input splits and reduce summary steps. If you have the Ambari TEZ View installed, then I would recommend inspecting each of the TEZ tasks and look at the SHUFFLE BYTES counters to see how much data is moving between the steps. If you see the early steps are moving a lot of data between the tasks then you have probably found the root cause of your out of memory exception and you should be able to tune your Hive query to filter data earlier in the p

[#7]

Where exactly is the OOM occurring? Is it in the AM? The table are partitioned?

Does this work for you with a scaled down dataset? Table definition will be helpful yo look at

[#8]

I am facing the same issue . Can someone please help