Support Questions
Find answers, ask questions, and share your expertise

Yarn resource waste with dynamic allocation enabled

Explorer

with dynamic allocation enabled, does yarn avoid resource waste?

 

assume a spark job requests:

--num-executors 50

--executor-memory 64gb

--executor-cores 10

 

but at runtime, the job only uses 10 executors, 2gb memory and 2 cores per executor.

does the resource manager reclaim the waste?