Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Map and Reduce tasks heap size

Map and Reduce tasks heap size

New Contributor

Hi,

 

We are using Cloudera Express 5.4.8.

 

Latetly we are facing an issue with passing heap size argguments for the map and reduce tasks, and Isuspect that it's related to client overide configuration.

 

We are running our mapreduce job with "hadoop jar" and passing JVM arguments on the command:

 

 -Dmapreduce.map.java.opts=-Xmx1700m -Dmapreduce.reduce.java.opts=-Xmx2200m.

 

However, it seems that these are not passed to the child JVMs, and instead it uses the deafult java heap size.

 

I think the reason for this is the "Map Task Maximum Heap Size (Client Override)" and "Reduce Task Maximum Heap Size (Client Override)". I'm trying to put 0 into those, but then I get an error on Cloudera Manager web UI "0 GiB is less than the minimum allowed value 32 MiB" and  the JT fails to start. How do I disable this configuration?

 

Any directions will be greatly appreciated.

 

Assaf

 

 

2 REPLIES 2

Re: Map and Reduce tasks heap size

New Contributor

Hi Again,

 

I think I made some progress here.

Vieweing mapred-site.xml of one of the tt (fromCloudera Manager > Cluster > HDFS > Instances > tt01 > Processes see here), I see entries the java.opts entries has a <final>true</final> (see snippet below).

I can't seem to get rid of those through Cloudera manager.

 

Any help will be very much appriciated.

 

Thanks,

Assaf

 

  <property>
    <name>mapred.child.java.opts</name>
    <value></value>
    <final>true</final>
  </property>
  <property>
    <name>mapred.map.child.java.opts</name>
    <value> -Xmx2147483648</value>
    <final>true</final>
  </property>
  <property>
    <name>mapred.reduce.child.java.opts</name>
    <value> -Xmx3221225472</value>
    <final>true</final>
  </property>

 

 

Highlighted

Re: Map and Reduce tasks heap size

Explorer

Hi,

 

you might try these options:

 

-Dmapreduce.map.java.opts.max.heap=<size_in_bytes> -Dmapreduce.map.memory.mb=<size_in_mb>

 

Just change the map to reduce for the same options.

 

Doni