Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How AM decides how many mapreduce jobs wiill be created for particular execution ? and in which memory the map task and reduce task will be performed ?

Solved Go to solution

How AM decides how many mapreduce jobs wiill be created for particular execution ? and in which memory the map task and reduce task will be performed ?

Expert Contributor

i want to know internal representation of YARN and MAPREDUCE. i am new to hadoop. i am getting how exactly the jobs get executed.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How AM decides how many mapreduce jobs wiill be created for particular execution ? and in which memory the map task and reduce task will be performed ?

Hi @heta desai

The Application Master will launch one MapTask for each map split. Typically, there is a map split for each input file. If the input file is too big (bigger than the HDFS block size) then we have two or more map splits associated to the same input file.

Also the memory used fro map and reduce task is RAM of Nodemanagers.

Please refer to it for more details -

http://ercoppa.github.io/HadoopInternals/AnatomyMapReduceJob.html

3 REPLIES 3

Re: How AM decides how many mapreduce jobs wiill be created for particular execution ? and in which memory the map task and reduce task will be performed ?

Hi @heta desai

The Application Master will launch one MapTask for each map split. Typically, there is a map split for each input file. If the input file is too big (bigger than the HDFS block size) then we have two or more map splits associated to the same input file.

Also the memory used fro map and reduce task is RAM of Nodemanagers.

Please refer to it for more details -

http://ercoppa.github.io/HadoopInternals/AnatomyMapReduceJob.html

Re: How AM decides how many mapreduce jobs wiill be created for particular execution ? and in which memory the map task and reduce task will be performed ?

Expert Contributor

Thank you.

Highlighted

Re: How AM decides how many mapreduce jobs wiill be created for particular execution ? and in which memory the map task and reduce task will be performed ?

Can you please accept my answer if it answered your question ? :)