https://community.cloudera.com/t5/Support-Questions/How-Yarn-distributes-containers-on-nodemanagers-while/m-p/229929#M191782 <P><A rel="user" href="https://community.cloudera.com/users/13501/rohitkhose.html" nodeid="13501">@Rohit Khose</A> YARN is resource negotiator for your cluster. Spark ( like other hadoop application) requests YARN for resources specified by the user and if available it will use them.</P><P>You can enable spark dynamic allocation so the spark application can scale up/down executors depending on the need.</P><P><A href="https://spark.apache.org/docs/1.6.1/configuration.html#dynamic-allocation" target="_blank">https://spark.apache.org/docs/1.6.1/configuration.html#dynamic-allocation</A></P> Tue, 29 Aug 2017 15:07:41 GMT SunilS 2017-08-29T15:07:41Z