Reply
New Contributor
Posts: 1
Registered: ‎05-10-2017

Out of memory on the spark executor when submitting using Oozie

currently we are trying to send input between 2g-4g and our spark executor settings are below

 

  <spark-opts>--num-executors 4 --executor-cores 4 --executor-memory 9g --driver-memory 9g</spark-opts>

 

We are getting Outof memory on the data node executor  with these settings. 

 

app flow:

python script kicks Oozie -> spark

 

Data node size : 8 physcial cores and 32gb RAM

 

ANy idea what need to be the the right executor settings for this sized physcial node

 

 

 

 

 

Champion
Posts: 462
Registered: ‎05-16-2016

Re: Out of memory on the spark executor when submitting using Oozie

Could you please share the whole error  stack trace 

Explorer
Posts: 18
Registered: ‎06-13-2017

Re: Out of memory on the spark executor when submitting using Oozie

hi, you need to increase your YARN container memory settings in CM. it has to be bigger than your --executor-memory.
Announcements