Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn can use 24GB of ram, but gives oom with only 12GB used

Highlighted

Yarn can use 24GB of ram, but gives oom with only 12GB used

Expert Contributor

Hello,

I am running on hdp 2.6 on QWS with 8 nodes.

Here are my relevant settings:

  • yarn.nodemanager.resource.memory-mb=24gb
  • yarn.scheduler.maximum-allocation-mb=24gb
  • yarn.scheduler.minimum-allocation-mb=1gb
  • tez container size=4gb

I have 32GB per DN, I see in my monitoring that only 13GB (max) is used, but I still receive when running a big hive query:

received a TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)

Could you help me make sense of this?

Thanks