https://community.cloudera.com/t5/Support-Questions/Spark-driver-memory-keeps-growing/m-p/165012#M127379 <P>HI Pierre,</P><P>We would need to look at the code.</P><P>Can you a do a persist just before stage 63 and before stage 65 check the spark UI storage tab and executor tab for data skew. If there is data skew, you will need to add a salt key to your key. </P><P>You could also look at creating a dataframe from the RDD rdd.toDF() and apply UDF on it. DF manage memory more efficiently.</P><P>Best,</P><P>Amit</P> Tue, 09 Aug 2016 14:35:26 GMT anandi 2016-08-09T14:35:26Z