question Re: Jobs fail in Yarn with out of Java heap memory error in Support Questions

Jobs fail in Yarn with out of Java heap memory error

IT.Services — Fri, 16 Sep 2022 09:07:42 GMT

We are running Yarn on CDH 5.1 with 14 nodes using 6 GB of memory. I understand this is not a log of memory, but it is all we could put together. Most jobs complete without error, but a few of the larger MapReduce jobs fail with an out of Java heap memory error. The jobs fail on a Reduce task that either sorts or groups data. We recently upgraded to CDH 5.1 from CDH 4.7 and ALL of these jobs succeeded on MapReduce v1. Looking in the logs I see that the Application has retired a few times before failing. Can you see anything wrong with the way the resources are configured?

Java Heap Size of NodeManager in Bytes	1 GB
yarn.nodemanager.resource.memory-mb	6 GB
yarn.scheduler.minimum-allocation-mb	1 GB
yarn.scheduler.maximum-allocation-mb	6 GB
yarn.app.mapreduce.am.resource.mb	1.5 GB
yarn.nodemanager.container-manager.thread-count	20
yarn.resourcemanager.resource-tracker.client.thread-count	20
mapreduce.map.memory.mb	1.5 GB
mapreduce.reduce.memory.mb	3 GB
mapreduce.map.java.opts	"-Djava.net.preferIPv4Stack=true -Xmx 1228m";
mapreduce.reduce.java.opts	"-Djava.net.preferIPv4Stack=true -Xmx2457m";
mapreduce.task.io.sort.factor	5
mapreduce.task.io.sort.mb	512 MB
mapreduce.job.reduces	2
mapreduce.reduce.shuffle.parallelcopies	4

One thing that might help, Yarn runs 4 containers per node, can this be reduced?

Re: Jobs fail in Yarn with out of Java heap memory error

bcwalrus — Mon, 15 Sep 2014 16:33:12 GMT

What are your MR1 settings? Do reducers used to get -Xmx2457m on MR1?

Also, the AM memory at 1.5GB is a bit high. You could probably cut that to 1GB.

Re: Jobs fail in Yarn with out of Java heap memory error

IT.Services — Mon, 15 Sep 2014 17:30:45 GMT

Thanks bcwalrus, very good question:

In MRv1, we configured the Java Heap Size of TaskTracker in Bytes with: 600 MB. Do you think I've set this too high in MRv2?

I'll cut the AM memory down to 1 GB, that is good advice. That will save me some memory on the node.

Kevin

Re: Jobs fail in Yarn with out of Java heap memory error

bcwalrus — Tue, 16 Sep 2014 00:25:50 GMT

I'm not asking about the heap of the TT process. I'm asking about the -Xmx of the reducers of this particular job (which used to work in MR1 and is failing in MR2).

You said that the reducers are failing due to OOME. They're getting 2457MB in MR2. What did they get in MR1?

Re: Jobs fail in Yarn with out of Java heap memory error

IT.Services — Tue, 16 Sep 2014 15:32:30 GMT

I don't think we ever changed the -Xmx on the reducers in MR1, this would have remained the default. Do you know what the default is for MR1?

Re: Jobs fail in Yarn with out of Java heap memory error

bcwalrus — Tue, 16 Sep 2014 16:11:08 GMT

The default in MR1 is unlimited, for both mapred.cluster.max.reduce.memory.mb and mapred.job.reduce.memory.mb. What did you set for mapred.child.java.opts (MR1)? Do you have the job counters from a big MR1 job? It'll tell you the average memory usage across the reducers, which will give you a good idea on what to set for MR2.

Re: Jobs fail in Yarn with out of Java heap memory error

IT.Services — Tue, 16 Sep 2014 16:36:55 GMT

Thanks for your help with this problem, I didn't know the default was unlimited. The max number of reducers for each TT was set at 2. I don't have the job counters from a big MR1 job, but I might be able to look them up. Where would I find them?

Re: Jobs fail in Yarn with out of Java heap memory error

IT.Services — Tue, 16 Sep 2014 17:23:38 GMT

From the Yarn logs I can see that Yarn believes that a huge amount of virtual memory is available before the job is killed, why is it using so much Virtual memory? Where is this set?

2014-09-16 10:18:30,803 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 51870 for container-id container_1410882800578_0001_01_000001: 797.0 MB of 2.5 GB physical memory used; 1.8 GB of 5.3 GB virtual memory used
2014-09-16 10:18:33,829 INFO 
...
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1410882800578_0005_01_000048
2014-09-16 10:18:34,431 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=admin	IP=192.168.210.251	OPERATION=Stop Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1410882800578_0005	CONTAINERID=container_1410882800578_0005_01_000048
2014-09-16 10:18:34,432 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1410882800578_0005_01_000048 transitioned from RUNNING to KILLING
2014-09-16 10:18:34,433 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1410882800578_0005_01_000048
2014-09-16 10:18:34,462 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1410882800578_0005_01_000048 is : 143
2014-09-16 10:18:34,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1410882800578_0005_01_000048 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
2014-09-16 10:18:34,553 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /space1/yarn/nm/usercache/admin/appcache/application_1410882800578_0005/container_1410882800578_0005_01_000048
2014-09-16 10:18:34,556 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /space2/yarn/nm/usercache/admin/appcache/application_1410882800578_0005/container_1410882800578_0005_01_000048
2014-09-16 10:18:34,558 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=admin	OPERATION=Container Finished - Killed	TARGET=ContainerImpl	RESULT=SUCCESS	APPID=application_1410882800578_0005	CONTAINERID=container_1410882800578_0005_01_000048

Re: Jobs fail in Yarn with out of Java heap memory error

bcwalrus — Tue, 16 Sep 2014 17:31:47 GMT

Virtual memory checking is pointless. Please make sure that `yarn.nodemanager.vmem-check-enabled' is turned off. The CDH default is off already.

That shouldn't matter though. You said that the job died due to OOME. It didn't die because it got killed by NM.

Re: Jobs fail in Yarn with out of Java heap memory error

IT.Services — Tue, 16 Sep 2014 17:38:04 GMT

Thanks bcwalrus, what if I increased the mapreduce.task.io.sort.factor, which is currently set to 5?

Also, do you know if it would be helpful to increase the mapreduce.reduce.java.opts.max.heap from the current setting of 787.69 MiB? Or is this not helpful?

Re: Jobs fail in Yarn with out of Java heap memory error

bcwalrus — Tue, 16 Sep 2014 17:56:36 GMT

I'm confused. Your initial post says that the reduce heap is 2457MB. Now it seems that's just 787.69MB. Which one is right? What does /etc/hadoop/conf/mapred-site.xml say?

Re: Jobs fail in Yarn with out of Java heap memory error

IT.Services — Tue, 16 Sep 2014 18:42:14 GMT

Here is what I have. Is the map/reduce java opts being overwritten, there are two entries?

<configuration>
<property>
<name>mapreduce.job.split.metainfo.maxsize</name>
<value>10000000</value>
</property>
<property>
<name>mapreduce.job.counters.max</name>
<value>120</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.compress</name>
<value>false</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.compress.type</name>
<value>BLOCK</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.compress.codec</name>
<value>org.apache.hadoop.io.compress.DefaultCodec</value>
</property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>zlib.compress.level</name>
<value>DEFAULT_COMPRESSION</value>
</property>
<property>
<name>mapreduce.task.io.sort.factor</name>
<value>5</value>
</property>
<property>
<name>mapreduce.map.sort.spill.percent</name>
<value>0.8</value>
</property>
<property>
<name>mapreduce.reduce.shuffle.parallelcopies</name>
<value>4</value>
</property>
<property>
<name>mapreduce.task.timeout</name>
<value>600000</value>
</property>
<property>
<name>mapreduce.client.submit.file.replication</name>
<value>4</value>
</property>
<property>
<name>mapreduce.job.reduces</name>
<value>2</value>
</property>
<property>
<name>mapreduce.task.io.sort.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.map.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.reduce.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.job.reduce.slowstart.completedmaps</name>
<value>0.8</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>blvdevhdp05.ds-iq.corp:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>blvdevhdp05.ds-iq.corp:19888</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>1536</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
<value>1</value>
</property>
<property>
<name>mapreduce.job.ubertask.enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Djava.net.preferIPv4Stack=true -Xmx825955249</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Djava.net.preferIPv4Stack=true -Xmx768m -Xmx825955249</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Djava.net.preferIPv4Stack=true -Xmx1280m -Xmx825955249</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1280</value>
</property>
<property>
<name>mapreduce.map.cpu.vcores</name>
<value>1</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1792</value>
</property>
<property>
<name>mapreduce.reduce.cpu.vcores</name>
<value>1</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH</value>
</property>
<property>
<name>mapreduce.admin.user.env</name>
<value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value>
</property>
<property>
<name>mapreduce.shuffle.max.connections</name>
<value>80</value>
</property>
</configuration>

Re: Jobs fail in Yarn with out of Java heap memory error

IT.Services — Tue, 16 Sep 2014 19:05:59 GMT

I have an easy question, if I increase mapreduce.reduce.shuffle.parallelcopies from 4 to 10, will that increase or decrease memory used by the node?

It seems to me that if this is increased that data is written quickly to files and out of memory. But I might be wrong...

Re: Jobs fail in Yarn with out of Java heap memory error

bcwalrus — Wed, 17 Sep 2014 17:12:47 GMT

<name>mapreduce.reduce.java.opts</name>
<value>-Djava.net.preferIPv4Stack=true -Xmx1280m -Xmx825955249</value>

Limits the heap to ~825MB. Most JVMs resolve duplicate args by picking the last one. So this is nowhere close to the 3GB that you intended. You should find out where you set this in CM and change it.

Do that before you play with parallelcopies. But to answer your questions, yes, it'll increase CPU, memory & network usage. And it could lead to more disk spills and slow down your job.

Re: Jobs fail in Yarn with out of Java heap memory error

IT.Services — Wed, 17 Sep 2014 18:25:02 GMT

You pointed out the problem and I removed the -Xmx825955249 from where I had entered it in Cloudera Manager. I was using the wrong field to update the value. Thank you so much for sticking with me and helping me resolve this issue! The jobs now succeed!

Kevin Verhoeven