Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

SOLVED Go to solution

How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

New Contributor

As per my understanding spark does not need to be installed on all the node in a yarn cluster. Spark installation is only required at the node(usually gateway node) from where spark-submit script is fired.

 

As per spark programming guide

To make Spark runtime jars accessible from YARN side, you can specify spark.yarn.archive or spark.yarn.jars.

How does libraries containing Spark code (i.e spark runtime jar available in ../spark-2.0.1-bin-hadoop2.6/jars) get distributed to Physical Worker Node(where executor are launched) in a YARN cluster.

 

Does this libraries gets copied to worker node every time we run a spark Application.

 

Thank You.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

Expert Contributor

When running on Spark, the spark archive gets distributed to worker nodes via the ContainerLocalizer (aka distributed cache).  Spark first uploads files to HDFS and then worker nodes can handle downloading the jar when needed from HDFS.  The localizer has some checks to only download the jar when it has changed or has been removed from the worker, so it can reuse the jar and not have to download it again if it still exists lo

3 REPLIES 3

Re: How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

Expert Contributor

When running on Spark, the spark archive gets distributed to worker nodes via the ContainerLocalizer (aka distributed cache).  Spark first uploads files to HDFS and then worker nodes can handle downloading the jar when needed from HDFS.  The localizer has some checks to only download the jar when it has changed or has been removed from the worker, so it can reuse the jar and not have to download it again if it still exists lo

Re: How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

New Contributor

As you mentioned "The localizer has some checks to only download the jar when it has changed or has been removed from the worker".

Is there any similar checks happens while copying spark jar from node (from where spark application is launched) to HDFS. Mean to say for multiple Spark Application launch spark jar will be copied to HDFS only once.

 

Thank You.  

 

Re: How does spark runtime jar (../spark-2.0.1/jars) get distributed to Physical Worker Node

Expert Contributor

When spark determines it needs to use yarn's localizer, it will always load the jar to HDFS, it does not attempt to check if the file changed before loading.

 

When using the Spark distributed included with CDH, the spark jar is already loaded to all nodes and specifies the jar is local.  When specifying it is local, spark will not upload the jar and yarn's localizer is not used.