Just starting to understand Spark memory management on yarn and got few questions that I thought would be better to ask experts here. 1. Is there a way to restrict max size that users can use for Spark executor and Driver when submitting jobs on Yarn cluster? 2. What the best practice around determining number of executor required for a job? Is there a max limit that users can be restricted to? 3. How RM handles resource allocation if most of the resources are consumed by Spark jobs in a queue? How preemption is handled? ... View more