Support Questions
Find answers, ask questions, and share your expertise

Yarn resource waste with dynamic allocation enabled

Yarn resource waste with dynamic allocation enabled

Explorer

with dynamic allocation enabled, does yarn avoid resource waste?

 

assume a spark job requests:

--num-executors 50

--executor-memory 64gb

--executor-cores 10

 

but at runtime, the job only uses 10 executors, 2gb memory and 2 cores per executor.

does the resource manager reclaim the waste?