currently we are trying to send input between 2g-4g and our spark executor settings are below
<spark-opts>--num-executors 4 --executor-cores 4 --executor-memory 9g --driver-memory 9g</spark-opts>
We are getting Outof memory on the data node executor with these settings.
python script kicks Oozie -> spark
Data node size : 8 physcial cores and 32gb RAM
ANy idea what need to be the the right executor settings for this sized physcial node
Could you please share the whole error stack trace