https://community.cloudera.com/t5/Support-Questions/completed-containers-does-not-freed-Yarn-resources-not-freed/m-p/236723#M198536 <P>For this you need to use the spark dynamic allocation.</P><P><STRONG>Dynamic Allocation (of Executors)</STRONG> (aka <EM>Elastic Scaling</EM>) is a Spark feature that allows for adding or removing <A href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-Executor.html">Spark executors</A> dynamically to match the workload.</P><P>Unlike the "traditional" static allocation where a Spark application reserves CPU and memory resources upfront (irrespective of how much it may eventually use), in dynamic allocation you get as much as needed and no more. It scales the number of executors up and down based on workload, i.e. idle executors are removed, and when there are pending tasks waiting for executors to be launched on, dynamic allocation requests them</P> Fri, 29 Mar 2019 11:43:22 GMT chennuri_gouris 2019-03-29T11:43:22Z