question Re: How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node in Archives of Support Questions (Read Only)

How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

dkdeepak — Fri, 16 Sep 2022 10:50:25 GMT

As per my understanding spark does not need to be installed on all the node in a yarn cluster. Spark installation is only required at the node(usually gateway node) from where spark-submit script is fired.

As per spark programming guide

To make Spark runtime jars accessible from YARN side, you can specify spark.yarn.archive or spark.yarn.jars.

How does libraries containing Spark code (i.e spark runtime jar available in ../spark-2.0.1-bin-hadoop2.6/jars) get distributed to Physical Worker Node(where executor are launched) in a YARN cluster.

Does this libraries gets copied to worker node every time we run a spark Application.

Thank You.

Re: How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

hubbarja — Tue, 13 Dec 2016 03:48:53 GMT

When running on Spark, the spark archive gets distributed to worker nodes via the ContainerLocalizer (aka distributed cache). Spark first uploads files to HDFS and then worker nodes can handle downloading the jar when needed from HDFS. The localizer has some checks to only download the jar when it has changed or has been removed from the worker, so it can reuse the jar and not have to download it again if it still exists lo

Re: How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

dkdeepak — Wed, 14 Dec 2016 14:40:51 GMT

As you mentioned "The localizer has some checks to only download the jar when it has changed or has been removed from the worker".

Is there any similar checks happens while copying spark jar from node (from where spark application is launched) to HDFS. Mean to say for multiple Spark Application launch spark jar will be copied to HDFS only once.

Thank You.

Re: How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

hubbarja — Thu, 15 Dec 2016 03:10:08 GMT

When spark determines it needs to use yarn's localizer, it will always load the jar to HDFS, it does not attempt to check if the file changed before loading.

When using the Spark distributed included with CDH, the spark jar is already loaded to all nodes and specifies the jar is local. When specifying it is local, spark will not upload the jar and yarn's localizer is not used.