question Re: Spark Application on YARN . in Archives of Support Questions (Read Only)

Spark Application on YARN .

parameswarnc — Wed, 15 Mar 2017 18:31:46 GMT

Hi All,

When executing the spark application on YARN cluster can I access the local file system (Underlying OS FS).

Though YARN is pointing to HDFS .

Thanks ,

Param.

sandyy006 — Wed, 15 Mar 2017 19:56:19 GMT

Yes, you can access the local file. Here is the sample:-

spark-shell --master yarn-client

scala> sc.textFile("file:///etc/passwd").count()

res0: Long = 40

parameswarnc — Wed, 15 Mar 2017 22:24:36 GMT

Thanks its worked ..I tried this already but forgot to create the file on each node ,now its fine.

And I just got one more question here :

If I run in spark app YARN mode I can set the memory parameter through sparkconfiguration using spark.yarn.driver.memoryOverhead properties ,

Is something similar available for the standalone and local mode ?

Thanks in advance ,

Param.

sandyy006 — Thu, 16 Mar 2017 00:35:08 GMT

@Param NC , Please close this thread by accepting the answer and consider asking new question.