https://community.cloudera.com/t5/Support-Questions/spark-job-shuffle-write-super-slow/m-p/220401#M182286 <P>Hey <A rel="user" href="https://community.cloudera.com/users/11642/pradeepbill.html" nodeid="11642">@pradeep arumalla</A>!<BR />I'm not a specialist in coding or spark, but did you tried to change your <STRONG>groupByKey</STRONG> for <STRONG>reduceByKey</STRONG> (at lhe last line)?</P><P>And about the executors --num-executors, how are you launching your job, is it by spark-submit? Could you share with us?</P><P><BR />BTW: here's some links about shuffling <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span> <BR /><A href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-shuffle.html" target="_blank">https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-shuffle.html</A><BR /><A href="https://0x0fff.com/spark-architecture-shuffle/" target="_blank">https://0x0fff.com/spark-architecture-shuffle/</A></P><P>Hope this helps!</P> Tue, 12 Jun 2018 22:14:22 GMT vmurakami 2018-06-12T22:14:22Z