question Re: [TEZ] are partition, sort and shuffle built-in ? in Support Questions

[TEZ] are partition, sort and shuffle built-in ?

sebastien_frack — Thu, 28 Dec 2017 00:37:03 GMT

Hello,

If the concept of MapReduce is pretty clear in my mind, i can't say so much for Tez.

MapReduce performs its work through Map > Partition, Sort, Shuffle > Reduce, and I know well each of these phases...

But for Tez, and more precisely, between two Vertices (considering a Vertices Map and a Vertices Reduce), how is it ?

Is there a built-in "partition, sort, shuffle" like in MR ? Or is it to us to manage this internal logic (i read a word count example, it seems it is, but I prefer to be sure) ?

Thanks !

Re: [TEZ] are partition, sort and shuffle built-in ?

balavignesh_nag — Thu, 28 Dec 2017 14:13:19 GMT

@Sebastien F

Background execution of tez and mr has many similarities. Differences lies in the where the data are in placed to transform it. Tez uses DAG to process the data whereas mr doesn't use DAG.

This link would answer your question. Hope it helps!!

Re: [TEZ] are partition, sort and shuffle built-in ?

sebastien_frack — Mon, 01 Jan 2018 19:52:08 GMT

thanks @Bala Vignesh N V ; it helps 🙂