https://community.cloudera.com/t5/Support-Questions/Executing-Spark-submit-with-yarn-cluster-mode-and-got-OOM-in/m-p/166903#M129240 <P>Spark allocates memory based on option parameters, which can be passed in multiple ways: </P><P>1) via the command-line (as you do)</P><P>2) via programmatic instructions</P><P>3) via the "spark-defaults.conf" file in the "conf" directory under your $SPARK_HOME</P><P>Second, there are separate config params for the driver and the executors. This is important, because the main difference between "yarn-client" and "yarn-cluster" mode is where the Driver lives (either on the client, or on cluster within the AppMaster). Therefore, we should look at your driver config parameters.</P><P>It looks like these are your driver-related options from the command-line:</P><PRE>--driver-memory 5000m --driver-cores 2 --conf spark.yarn.driver.memoryOverhead=1024 --conf spark.driver.maxResultSize=5g --driver-java-options "-XX:MaxPermSize=1000m"</PRE><P>It is possible that the AppMaster is running on a node that does not have enough memory to support your option requests, e.g. that the sum of driver-memory (5G) and PermSize (1G), plus overhead (1G) does not fit on the node. I would try lowering the --driver-memory by 1G steps until you no longer get the OOM error.</P> Thu, 14 Apr 2016 02:07:18 GMT phargis 2016-04-14T02:07:18Z