About dkdeepak

dkdeepak · ‎12-14-2016

As you mentioned "The localizer has some checks to only download the jar when it has changed or has been removed from the worker". Is there any similar checks happens while copying spark jar from node (from where spark application is launched) to HDFS. Mean to say for multiple Spark Application launch spark jar will be copied to HDFS only once. Thank You.

dkdeepak · ‎12-08-2016

As per my understanding spark does not need to be installed on all the node in a yarn cluster. Spark installation is only required at the node(usually gateway node) from where spark-submit script is fired. As per spark programming guide To make Spark runtime jars accessible from YARN side, you can specify spark.yarn.archive or spark.yarn.jars. How does libraries containing Spark code (i.e spark runtime jar available in ../spark-2.0.1-bin-hadoop2.6/jars) get distributed to Physical Worker Node(where executor are launched) in a YARN cluster. Does this libraries gets copied to worker node every time we run a spark Application. Thank You.

Online	Offline
Last Visited	‎02-27-2017 03:15 AM

Member Since	‎11-14-2016 10:46 PM
Last Visited	‎02-27-2017 03:15 AM
Posts	3

Cloudera Community

Re: How does spark runtime jar (../spark-2.0.1/jar...

How does spark runtime jar (../spark-2.0.1/jars) g...