question How Reducers know where the mapper results are stored in Archives of Support Questions (Read Only)

How Reducers know where the mapper results are stored

gsrao_cse — Thu, 14 Jul 2016 21:36:54 GMT

1. How Reducers know where the mapper results are stored

Re: How Reducers know where the mapper results are stored

rajkumar_singh — Thu, 14 Jul 2016 22:00:38 GMT

on very high level,once map task get complete it notify Application master through heartbeat, AM keeps track mapping between map output and hosts. Reducer polla AM for map output locations untill it get all.

Re: How Reducers know where the mapper results are stored

sunile_manjee — Thu, 14 Jul 2016 23:03:25 GMT

I use this blog often when I forget the data movement between map-->reduce

The map outputs are copied to the reduce task JVM’s memory if they are small enough (the buffer’s size is controlled by mapred.job.shuffle.input.buffer.percent, which specifies the proportion of the heap to use for this purpose); otherwise, they are copied to disk. When the in-memory buffer reaches a threshold size (controlled by mapred.job.shuffle.merge.percent) or reaches a threshold number of map outputs(mapred.inmem.merge.threshold), it is merged and spilled to disk.