question Re: in which memory Map and Reduce tasks is performed ? in Support Questions

in which memory Map and Reduce tasks is performed ?

heta_desai — Wed, 29 Mar 2017 12:05:03 GMT

i need detail information on in which memory map and reduce task will be performed ? the reduce will bring all the map task's output into one node and than performs reduce and give final output ?

Re: in which memory Map and Reduce tasks is performed ?

jsensharma — Wed, 29 Mar 2017 12:28:29 GMT

@heta desai The following doc will give mode detail on this:

1. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_command-line-installation/content/determine-hdp-memory-config.html

2. https://hortonworks.com/apache/mapreduce/#section_1

3. https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

You can specify the minimum unit of RAM to allocate for a Container. The tasks are run within containers launched by YARN. mapreduce.{map|reduce}.memory.mb is used by YARN to set the memory size of the container being used to run the map or reduce task. If the task grows beyond this limit, YARN will kill the container.

Re: in which memory Map and Reduce tasks is performed ?

shivkumar82015 — Wed, 29 Mar 2017 14:30:42 GMT

Its takes memory from datanode and same Node Manager, where map split is stored(Due to data locality) and map output is stored in an in-memory buffer.

when this buffer is almost full then we start (in parallel) the spilling phase in order to remove data from it and reducer output will be stored on the local filesystem.

Re: in which memory Map and Reduce tasks is performed ?

heta_desai — Thu, 30 Mar 2017 14:47:02 GMT

if data is distributed over 3 nodes. the final result set will be merge of this 3 node data. so my confusion is where this merge operation will be perform ?