https://community.cloudera.com/t5/Support-Questions/Spark-driver-memory-keeps-growing/m-p/165006#M127373 <P>Broadcast is done with blocks of data. See the spark.broadcast.blockSize property <A target="_blank" href="https://spark.apache.org/docs/1.6.1/configuration.html">here</A>. This explains why the value grows in the log output.</P><P>How big is the file you are broadcasting? You can use the <A target="_blank" href="https://spark.apache.org/docs/1.6.1/api/java/index.html?org/apache/spark/util/SizeEstimator.html">SizeEstimator</A> to get a sense of what your object will really occupy. Then make sure your "--driver-memory" and "--executor-memory" has enough breathing room. Guidance for tuning can be found here: <A href="https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/ch_tuning-spark.html" target="_blank">https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/ch_tuning-spark.html</A> </P> Thu, 04 Aug 2016 21:13:44 GMT clukasik 2016-08-04T21:13:44Z