https://community.cloudera.com/t5/Support-Questions/How-to-get-SparkContext-in-executor/m-p/114171#M76969 <P><A rel="user" href="https://community.cloudera.com/users/420/vjain.html" nodeid="420">@Vedant Jain</A> , <A rel="user" href="https://community.cloudera.com/users/452/bwalter.html" nodeid="452">@Bernhard Walter</A> </P><P> well, may be my question is misleading you , let me elaborate it.</P><PRE>val textFile = sc.textFile("hdfs://...") val counts = textFile.flatMap(line =&gt; line.split(" ")) .map(word =&gt; (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...")</PRE><P>a simple wordcount problem.. this piece of code given to driver program, which creates DAG and stages, and given task to respective worker nodes where actual operation is happening.</P><P>Now, lets look at the first line of the program. From the file, RDD is generated (SparkContext implemented textFile() function which generates RDD from file). file is resides in worker node. from worker node, we needs to get the RDD out.</P><P>In order to acheive that , Worker node ( or executor ) needs to have the SparkContext, Isn't it ?</P><P>My Question is, How does executor gets the spark context ?</P> Tue, 22 Nov 2016 11:16:01 GMT gobi_subramani 2016-11-22T11:16:01Z