Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spark job on yarn

spark job on yarn

What is benefit of launching spark job after setting --master to yarn ?

2 REPLIES 2

Re: spark job on yarn

--master takes two options local and yarn

--master spawns the executors on yarn as container , else your spark job will be run locally

Re: spark job on yarn

@Anurag Mishra There are two deploy modes that can be used to launch Spark applications on YARN. In cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. In client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.

ref: https://spark.apache.org/docs/latest/running-on-yarn.html for more details.