Additionally, what exactly does dynamic allocation mean?? (I know it means allocating containers/executors on the fly but please elaborate) What are "spark.dynamicAllocation.maxExecutors"?? What should its value be?
How are each of these parameters related to each other?? What should be the setting of all these parameters for best performance on the cluster?
I read the documentation but I am afraid I am still confused with the terminology. Really appreciate if someone can elaborate.