Map reduce Flow clarification

vamsi123 — Sat, 24 Nov 2018 19:37:21 GMT

Which one will occur first in MapReduce Flow among shuffling and sorting?

To my knowledge shuffling will occur first and then Sorting? Correct me I am wrong.

Any body can explain these two things?

Below statement from the Definative guide:

MapReduce makes the guarantee that the input to every reducer is sorted by key. The process by which the system performs the sort—and transfers the map outputs to the reducers as inputs—is known as the shuffle.

Re: Map reduce Flow clarification

vamsi123 — Mon, 26 Nov 2018 20:29:59 GMT

Hi experts

Anybody Input on my mail?

Re: Map reduce Flow clarification

jagadeesan — Sat, 17 Aug 2019 23:28:05 GMT

@vamsi valiveti

Shuffling is the process of transferring data from the mappers to the reducers, so I think it is obvious that it is necessary for the reducers, since otherwise, they wouldn't be able to have any input (or input from every mapper). Shuffling can start even before the map phase has finished, to save some time. That's why you can see a reduce status greater than 0% (but less than 33%) when the map status is not yet 100%.

Sorting saves time for the reducer, helping it easily distinguish when a new reduce task should start. It simply starts a new reduce task, when the next key in the sorted input data is different than the previous, to put it simply. Each reduce task takes a list of key-value pairs, but it has to call the reduce() method which takes a key-list(value) input, so it has to group values by key. It's easy to do so, if input data is pre-sorted (locally) in the map phase and simply merge-sorted in the reduce phase (since the reducers get data from many mappers).

A great source of information for these steps is this Yahoo tutorial.
A nice graphical representation of this is the following:

Note that shuffling and sorting are not performed at all if you specify zero reducers (setNumReduceTasks(0)). Then, the MapReduce job stops at the map phase, and the map phase does not include any kind of sorting (so even the map phase is faster) Ref

Please accept the answer you found most useful

question Re: Map reduce Flow clarification in Archives of Support Questions (Read Only)

Map reduce Flow clarification

Re: Map reduce Flow clarification

Re: Map reduce Flow clarification