@Sree Kupp I need some clarification. You want to set num_executors to 46, which is a maximum set, but you want to use all the capacity of the cluster which is 96, but you did not mention anything about the RAM allocated to each container. For example, if you have 96 cores and 4 x 128 = 516 GB (simplistically because some cores and memory need to be allocated to other processes running on your cluster), and your memory allocated per container (one core per container) is 8 GB that would require for 96 containers = 96 * 8 = 768. You are short 252 GB which is about round(252/8)=32 containers. That means, even you set num_executors to 96, you could possibly spin 64. What happened when you set num_executors to 96. On the other hand, even if you set it to 96 it does not mean anything that the maximum is 96. Spark can decide based on several factors, e.g. data locality, Regarding your 47 vs. 46 set, we need to investigate a bit more what the extra container does.
... View more
