Support Questions
Find answers, ask questions, and share your expertise

Is it possible to dynamically set map and reduce memory for a Mapreduce job?

 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

@Phoncy Joseph

The configs can be changed when you are submitting the job and not while the job is running. Below are the configs to change them while submitting the job.

-Dmapreduce.map.memory.mb=2048 -Dmapreduce.reduce.memory.mb=2048

View solution in original post

5 REPLIES 5

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

@Phoncy Joseph

The configs can be changed when you are submitting the job and not while the job is running. Below are the configs to change them while submitting the job.

-Dmapreduce.map.memory.mb=2048 -Dmapreduce.reduce.memory.mb=2048

View solution in original post

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

Thanks for the response. I wanted to know if the memory assignment could be done, without providing these while submitting jobs.

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

@Phoncy Joseph There are only 3 ways as of now:

- Global configs which are in mapred-site.xml

- Passing through command line while submitting the job

- Hardcode in the code(although this is not dynamic).

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

Thank you

Re: Is it possible to dynamically set map and reduce memory for a Mapreduce job?

@Phoncy Joseph You're welcome. Feel free to accept the answer if this helps you.