03-04-2019 02:49 PM
In Hive configuration we have
spark.dynamicAllocation.enabled = true, actually checked
spark.executor.cores = 6
Does dynamic override the set number of 6? It does not appear dynamic cores are being allocated.
03-07-2019 12:32 AM
Spark dynamic allocation allows spark to add and remove the executors dynamically based on your application's demand. But for each executor, it will still respect your settings (e.g. cores and RAM per executor) and use a unified and fixed spec.
Hope it helps.
03-07-2019 12:47 AM