Ok that was actually interesting so I had a look into the code. For open source projects always the definitive source: You can find most of it in the MapTask class.
-> Map Phase is run, output goes into Sorting Collector or DirectCollector ( latter if no reduce tasks )
-> The write already uses the partitioner, i.e. data is partitioned when going into the Sorting Collector -> Sorting Collector is a class called MapOutputBuffer -> In here we have a combiner class and a Sorting class. Data is sorted in a memory buffer and then spilled. -> First data is sorted using the sort buffer then written to disc either directly OR using the CombinerRunner and then writing it.
Combiner is also used in the Merge phase when spilled files are merged together into one output file
So the correct answer is They do not happen "after" each other, they happen together. While output data is written it is partitioned sorted, spilled and combined, merged and combined again.
Hope that helps. If you want more information just have a look into the code yourself. Its quite readable.
Ben PS: So we had 5 Datanodes running map tasks. Which node does partition happens & how many partitions will be created? There is one partitioned output file for each Map Task. So you have 10 Map Task and 2 Reducers. This means that there will be 2 output files for each Map Task one for each reducer (*). Number partitions = number reducers ( for each map task ) When the reducer spins up he starts downloading the output file for his partition from every map task as they finish. And merge sorts it into one input set. In your example Both reducer each will pull 10 datasets ( one from each map task) and merge sort them into a single valid input set. (*)actually each map task only writes one file with offsets to not create too many small files if I am not mistaken but that doesn't change the basic functionality.
... View more
