Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Map reduce Flow clarification

Solved Go to solution
Highlighted

Map reduce Flow clarification

Contributor

Which one will occur first in MapReduce Flow among shuffling and sorting?

To my knowledge shuffling will occur first and then Sorting? Correct me I am wrong.

Any body can explain these two things?

Below statement from the Definative guide:

MapReduce makes the guarantee that the input to every reducer is sorted by key. The process by which the system performs the sort—and transfers the map outputs to the reducers as inputs—is known as the shuffle.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Map reduce Flow clarification

Expert Contributor

@vamsi valiveti

Shuffling is the process of transferring data from the mappers to the reducers, so I think it is obvious that it is necessary for the reducers, since otherwise, they wouldn't be able to have any input (or input from every mapper). Shuffling can start even before the map phase has finished, to save some time. That's why you can see a reduce status greater than 0% (but less than 33%) when the map status is not yet 100%.

Sorting saves time for the reducer, helping it easily distinguish when a new reduce task should start. It simply starts a new reduce task, when the next key in the sorted input data is different than the previous, to put it simply. Each reduce task takes a list of key-value pairs, but it has to call the reduce() method which takes a key-list(value) input, so it has to group values by key. It's easy to do so, if input data is pre-sorted (locally) in the map phase and simply merge-sorted in the reduce phase (since the reducers get data from many mappers).

A great source of information for these steps is this Yahoo tutorial.
A nice graphical representation of this is the following:

93380-mr.png

Note that shuffling and sorting are not performed at all if you specify zero reducers (setNumReduceTasks(0)). Then, the MapReduce job stops at the map phase, and the map phase does not include any kind of sorting (so even the map phase is faster) Ref

Please accept the answer you found most useful

2 REPLIES 2

Re: Map reduce Flow clarification

Contributor

Hi experts

Anybody Input on my mail?

Re: Map reduce Flow clarification

Expert Contributor

@vamsi valiveti

Shuffling is the process of transferring data from the mappers to the reducers, so I think it is obvious that it is necessary for the reducers, since otherwise, they wouldn't be able to have any input (or input from every mapper). Shuffling can start even before the map phase has finished, to save some time. That's why you can see a reduce status greater than 0% (but less than 33%) when the map status is not yet 100%.

Sorting saves time for the reducer, helping it easily distinguish when a new reduce task should start. It simply starts a new reduce task, when the next key in the sorted input data is different than the previous, to put it simply. Each reduce task takes a list of key-value pairs, but it has to call the reduce() method which takes a key-list(value) input, so it has to group values by key. It's easy to do so, if input data is pre-sorted (locally) in the map phase and simply merge-sorted in the reduce phase (since the reducers get data from many mappers).

A great source of information for these steps is this Yahoo tutorial.
A nice graphical representation of this is the following:

93380-mr.png

Note that shuffling and sorting are not performed at all if you specify zero reducers (setNumReduceTasks(0)). Then, the MapReduce job stops at the map phase, and the map phase does not include any kind of sorting (so even the map phase is faster) Ref

Please accept the answer you found most useful

Don't have an account?
Coming from Hortonworks? Activate your account here