Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is it possible to dynamically set map and reduce memory for a Mapreduce job?

Solved Go to solution
Highlighted

Is it possible to dynamically set map and reduce memory for a Mapreduce job?

New Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

@Phoncy Joseph

The configs can be changed when you are submitting the job and not while the job is running. Below are the configs to change them while submitting the job.

-Dmapreduce.map.memory.mb=2048 -Dmapreduce.reduce.memory.mb=2048
5 REPLIES 5

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

@Phoncy Joseph

The configs can be changed when you are submitting the job and not while the job is running. Below are the configs to change them while submitting the job.

-Dmapreduce.map.memory.mb=2048 -Dmapreduce.reduce.memory.mb=2048

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

New Contributor

Thanks for the response. I wanted to know if the memory assignment could be done, without providing these while submitting jobs.

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

@Phoncy Joseph There are only 3 ways as of now:

- Global configs which are in mapred-site.xml

- Passing through command line while submitting the job

- Hardcode in the code(although this is not dynamic).

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

New Contributor

Thank you

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

@Phoncy Joseph You're welcome. Feel free to accept the answer if this helps you.