How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

[#1]

As per my understanding spark does not need to be installed on all the node in a yarn cluster. Spark installation is only required at the node(usually gateway node) from where spark-submit script is fired.

As per spark programming guide

To make Spark runtime jars accessible from YARN side, you can specify spark.yarn.archive or spark.yarn.jars.

How does libraries containing Spark code (i.e spark runtime jar available in ../spark-2.0.1-bin-hadoop2.6/jars) get distributed to Physical Worker Node(where executor are launched) in a YARN cluster.

Does this libraries gets copied to worker node every time we run a spark Application.

Thank You.

[#2 · Accepted solution]

When running on Spark, the spark archive gets distributed to worker nodes via the ContainerLocalizer (aka distributed cache).  Spark first uploads files to HDFS and then worker nodes can handle downloading the jar when needed from HDFS.  The localizer has some checks to only download the jar when it has changed or has been removed from the worker, so it can reuse the jar and not have to download it again if it still exists lo

[#3]

As you mentioned "The localizer has some checks to only download the jar when it has changed or has been removed from the worker".

Is there any similar checks happens while copying spark jar from node (from where spark application is launched) to HDFS. Mean to say for multiple Spark Application launch spark jar will be copied to HDFS only once.

Thank You.  

[#4]

When spark determines it needs to use yarn's localizer, it will always load the jar to HDFS, it does not attempt to check if the file changed before loading.

When using the Spark distributed included with CDH, the spark jar is already loaded to all nodes and specifies the jar is local.  When specifying it is local, spark will not upload the jar and yarn's localizer is not used.