https://community.cloudera.com/t5/Support-Questions/spark-dynamic-allocation-setting/m-p/104909#M67800 <P>Hi All,</P><P>I was testing spark dynamic resource allocation in spark. By default I see "spark-thrift-sparkconf.conf" contains all the dynamic allocation properties. But when I run the spark job "spark-shell --master yarn --num-executors 5 --executor-memory 3G", I expect it complain as I've requested number of executor in the job itself.</P><P>Then I modifed the custom spark-defaults.conf and added dynamic allocation properties:</P><PRE>spark.dynamicAllocation.enabled true spark.dynamicAllocation.initialExecutors 1 spark.dynamicAllocation.maxExecutors 5 spark.dynamicAllocation.minExecutors 1</PRE><P>And when I run the same job, I see below messages :</P><PRE>16/05/23 09:18:54 WARN SparkContext: Dynamic Allocation and num executors both set, thus dynamic allocation disabled. </PRE> <P>Also print below messages if needed more resources. My doubt is is dynamic allocation is defined by default? Which config we should define dynamic allocation properties?</P><PRE>6/05/23 09:39:47 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 4) 16/05/23 09:39:48 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 5) </PRE> Fri, 27 May 2016 17:53:06 GMT nyadav 2016-05-27T17:53:06Z