Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

in which memory Map and Reduce tasks is performed ?

avatar
Expert Contributor

i need detail information on in which memory map and reduce task will be performed ? the reduce will bring all the map task's output into one node and than performs reduce and give final output ?

1 ACCEPTED SOLUTION

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
3 REPLIES 3

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

Its takes memory from datanode and same Node Manager, where map split is stored(Due to data locality) and map output is stored in an in-memory buffer.

when this buffer is almost full then we start (in parallel) the spilling phase in order to remove data from it and reducer output will be stored on the local filesystem.

avatar
Expert Contributor

if data is distributed over 3 nodes. the final result set will be merge of this 3 node data. so my confusion is where this merge operation will be perform ?