Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Heap size configuration for map & reduce tasks

Heap size configuration for map & reduce tasks

New Contributor
 

Hi,

 

Not sure I posted this in the roght place so posting again here. 

We are using Cloudera Express 5.4.8.

 

Latetly we are facing an issue with passing heap size argguments for the map and reduce tasks, and Isuspect that it's related to client overide configuration.

 

We are running our mapreduce job with "hadoop jar" and passing JVM arguments on the command:

 

 -Dmapreduce.map.java.opts=-Xmx1700m -Dmapreduce.reduce.java.opts=-Xmx2200m.

 

However, it seems that these are not passed to the child JVMs, and instead it uses the deafult java heap size.

 

I think the reason for this is the "Map Task Maximum Heap Size (Client Override)" and "Reduce Task Maximum Heap Size (Client Override)". I'm trying to put 0 into those, but then I get an error on Cloudera Manager web UI "0 GiB is less than the minimum allowed value 32 MiB" and  the JT fails to start. How do I disable this configuration?

 

Any directions will be greatly appreciated.

 

Assaf

3 REPLIES 3

Re: Heap size configuration for map & reduce tasks

Master Guru
If you use YARN+MR2 these overrides do not apply. They do however apply to MR1 (JobTracker and TaskTracker).

To disable the config, unset the config. Click "Reset to default" inside the config field or hit the reset arrow (↩) visible in the config field. It will set the config back to "empty" state, meaning not applied.

Re: Heap size configuration for map & reduce tasks

New Contributor

Hi Harsh,

 

Thanks for the reply.

 

I tried these, but it sets the value to 0, validation error is thrown and the save button is disabled. See below.

 

This is the message beside the "Save" button:

=====================================

 

Please fix the validation error(s) to enable save

 

 

This is the message by the field (Map Task Maximum Heap Size (Client Override)):

===================================================================

 
0 GiB is less than the minimum allowed value 32 MiB.

 

 Any ideas?

 

Thanks,

Assaf

 

 

Re: Heap size configuration for map & reduce tasks

New Contributor

Hi Guys,

 

Meanwhile, we ran a few tests on our lab cluster. And found the "Final" entries in mapred-site.xml so that's what we did:

 

1. Logged into the embeded Postgres

2. Searched for entires with "Client Overide":

 

scm=> select * from configs WHERE attr LIKE '%override%';
 
3. Removed all (there were 2 rows)
4. Restarted Cloudera Manager
5. Deployed client configuration
6. Restarted mapreduce service.
 
Now, in the configuration tab the fields are empty, there are no "Final" entries in mapred-site.xml files, and it seems that the client configuration is passed as expected.
 
The question is did we screw up things or can we do the same on our production env.?
 
Obvioiusly I prefer a more gentle way to fix this.
 
Thanks
Assaf