https://community.cloudera.com/t5/Support-Questions/How-many-number-of-executors-will-be-created-for-a-spark/m-p/186505#M148607 <P><A rel="user" href="https://community.cloudera.com/users/16783/saranpons3.html" nodeid="16783">@Saravanan Selvam</A>, In yarn mode you can control the total number of executors needed for an application with --num-executor option. </P><P>However, if you do not explicitly specify --num-executor for spark application in yarn mode, it would typically start one executor on each Nodemanager. </P><P>Spark also has a feature called Dynamic resource allocation. It gives spark application a feature to dynamically scale the set of cluster resources allocated to your application up and down based on the workload. This way you can make sure that application is not over utilizing the resources. </P><P><A href="http://spark.apache.org/docs/1.2.0/job-scheduling.html#dynamic-resource-allocation" target="_blank">http://spark.apache.org/docs/1.2.0/job-scheduling.html#dynamic-resource-allocation</A></P> Wed, 05 Apr 2017 01:53:51 GMT yvora 2017-04-05T01:53:51Z