https://community.cloudera.com/t5/Support-Questions/YARN-memory-configuration-parameters-and-Java-Heap-Space/m-p/133022#M95692 <P>Hi,</P><P>I'm trying to configure the optimum memory configuration in YARN to implement some MR tasks in R. For the moment, I have a single node with around 40GB RAM available. I have tried different memory combinations but all of them result in Java Heap Space exceptions when trying to execute a simple MR R code (using the library <A href="https://github.com/RevolutionAnalytics/plyrmr/blob/master/pkg/tests/pipe.R">plyrmr</A>) to process a small (a few KB size) text file. The relevant memory configuration parameters I have so far (in yarn-site.xml and map-red.xml) are:</P><PRE>yarn.scheduler.maximum-allocation-mb = 24576 yarn.scheduler.minimum-allocation-mb = 3076 yarn.app.mapreduce.am.resource.mb = 3076 mapreduce.map.java.opts = -Xmx2457m mapreduce.map.memory.mb = 3072 mapreduce.reduce.java.opts = -Xmx4915m mapreduce.reduce.memory.mb = 6144 </PRE><P>Is there any other memory configuration parameter that needs to be set or adjusted? After launching the task, 2 split jobs are created and a Java Heap Space exception is raised. Looking through the YARN logs of the application that raises the exception, I stumple upon the following line after executing the launch_container.sh:</P><PRE>exec /bin/bash -c "$JAVA_HOME/bin/java -server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.0.0-169 -Xmx400M </PRE><P>What are these "400 MB" of Java Space for? I have checked a lot of different configuration files but I couldn't find any parameter related to these 400MB of space. Is there any other Java parameter that needs to be set in the aforementioned list of configuration properties?</P><P>The relevant log part of the MR task is:</P><PRE>INFO mapreduce.Job: Counters: 17 Job Counters Failed map tasks=7 Killed map tasks=1 Killed reduce tasks=1 Launched map tasks=8 Other local map tasks=6 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=37110 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=37110 Total time spent by all reduce tasks (ms)=0 Total vcore-seconds taken by all map tasks=37110 Total vcore-seconds taken by all reduce tasks=0 Total megabyte-seconds taken by all map tasks=114001920 Total megabyte-seconds taken by all reduce tasks=0 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0</PRE><P>Is there anything that I'm missing?</P><P>Thanks a lot for your time.</P> Mon, 18 Jul 2016 15:43:22 GMT jaimecarodrigue 2016-07-18T15:43:22Z