When spark determines it needs to use yarn's localizer, it will always load the jar to HDFS, it does not attempt to check if the file changed before loading.   When using the Spark distributed included with CDH, the spark jar is already loaded to all nodes and specifies the jar is local.  When specifying it is local, spark will not upload the jar and yarn's localizer is not used. ... View more