Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

​mapreduce.map.java.opts memory setting and configuration

​mapreduce.map.java.opts memory setting and configuration

New Contributor

I am going through several blogs and tutorials and everywhere I found the JVM heap size should be set to lower than the Map and Reduce memory defined. For example, suppose I have defined the following configuration in my mapred-site.xml file.

<name>mapreduce.map.memory.mb</name>
<value>4096</value>
<name>mapreduce.reduce.memory.mb</name>
<value>8192</value>

Then Each Container will run JVMs for the Map and Reduce tasks. The JVM heap size should be set to lower than the Map and Reduce memory defined above, so that they are within the bounds of the Container memory allocated by YARN. Therefore It should be something like this

mapreduce.map.java.opts=-Xmx3072m
mapreduce.reduce.java.opts=-Xmx6144m

Why cant be define the JVM heap size same as Mapper and Reducer memory size. Is there any pros and cons of this? What will happen if I define the JVM heap size same as Mapper and Reduce memory size.

1 REPLY 1
Highlighted

Re: ​mapreduce.map.java.opts memory setting and configuration

Hi @Sandeep SIngh,

Each mapper and reducer is a java process and we need some reserve memory to store the java code. Hence the Map/Reducer memory should be greater than the JVM heap size.