Hello, I need your advice regarding what seems like a strange behavior  in the cluster I'm using. Running Spark (2.4, Cloudera) on Yarn the configuration calls the set up of 10 executors:     spark = (SparkSession .builder.master("yarn") .config("spark.executor.cores", "12") .config("spark.executor.memory", "8G") .config("spark.num.executors", "10") .config("spark.driver.memory", "6G") .config("spark.yarn.am.memoryOverhead", "6G") .config("spark.executor.memoryOverhead", "5G") .config("spark.driver.memoryOverhead", "5G") .config("spark.sql.hive.convertMetastoreOrc", "true") .config("spark.executor.heartbeatInterval", "60s") .config("spark.network.timeout", "600s") .config("spark.driver.maxResultSize", "2g") .config("spark.driver.cores","4") .config("spark.executor.extraClassPath", "-Dhdp.version=current") .config("spark.debug.maxToStringFields", 200) .config("spark.sql.catalogImplementation", "hive") .config("spark.memory.fraction", "0.8") .config("spark.memory.storageFraction", "0.2") .config("spark.sql.hive.filesourcePartitionFileCacheSize", "0") .config("spark.yarn.maxAppAttempts", "10") .appName(app_name) .enableHiveSupport().getOrCreate())     However, the UI/ yarn logs show the start of only 2 executors, with a third starting at a later stage, to replace the second: 21/07/29 10:44:01 INFO Executor: Starting executor ID 3 on host <node name>   21/07/29 10:43:54 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 10.0 GB of 10 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.   The memory issues wouldn't have emerge had the application set the number of executors requested.    Can you think of any reason why the cluster may "choose" to reduce the number of executors to set up?   Any internal yarn/other configuration I should examine?    ... View more