Support Questions
Find answers, ask questions, and share your expertise

spark-submit memory optimization

Explorer

we are running diverse computational tasks mostly varying on time of execution and ram required .

 

for this we have the following args in spark-submit  :

  • --executor-cores {}  (usually 1)
  • --executor-memory {}  (varies  4GB-60GB
  • --driver-memory 8g 
  • --driver-cores 4
  • --num-executors 4

because we have limit ram we split our job for multiple grid submit for based on ram required for example 5 grid submits (4-10GB, 11-20GB, ...)

 

sometimes the ram we allocates is not enough (we allocated 20GB but needed 21GB) and the job failed. how i can set memory overflow for example (each job have 30% extra that is not dedicated for the container, or an extra 5GB that is not allocated)

in real life the server are at 50-70% ram capacity