Support Questions
Find answers, ask questions, and share your expertise

Out of memory on the spark executor when submitting using Oozie

Out of memory on the spark executor when submitting using Oozie

New Contributor

currently we are trying to send input between 2g-4g and our spark executor settings are below

 

  <spark-opts>--num-executors 4 --executor-cores 4 --executor-memory 9g --driver-memory 9g</spark-opts>

 

We are getting Outof memory on the data node executor  with these settings. 

 

app flow:

python script kicks Oozie -> spark

 

Data node size : 8 physcial cores and 32gb RAM

 

ANy idea what need to be the the right executor settings for this sized physcial node

 

 

 

 

 

2 REPLIES 2

Re: Out of memory on the spark executor when submitting using Oozie

Champion

Could you please share the whole error  stack trace 

Re: Out of memory on the spark executor when submitting using Oozie

Explorer
hi, you need to increase your YARN container memory settings in CM. it has to be bigger than your --executor-memory.