Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Dynamic Resource Allocation Executor Timeout

Spark Dynamic Resource Allocation Executor Timeout

Explorer

spark.dynamicAllocation.executorIdleTimeout = 60

 

Can this be lowered? Is there any benefit/harm?

 

My question is regarding preemption. Are executors of a preempted container "idle". or is this idle step skipped during preemption?

 

1 REPLY 1
Highlighted

Re: Spark Dynamic Resource Allocation Executor Timeout

Rising Star

Hello @regeamor

 

Thank you for posting the query with us.

 

Basically when you enable Dynamic allocation it gracefully remove the Idle containers which were idle for (60s as default) But when you lower the value will remove the executors frequently depending upon the executor's usage (just like the tasks getting allocated to those executors and the amount of tasks required)

 

https://spark.apache.org/docs/latest/job-scheduling.html#graceful-decommission-of-executors

 

Also, the above said behaviour is upto Spark (and yarn container pre-emption wont be aware of it)

Thanks,
Satz
Don't have an account?
Coming from Hortonworks? Activate your account here