https://community.cloudera.com/t5/Support-Questions/Executing-Spark-submit-with-yarn-cluster-mode-and-got-OOM-in/m-p/166901#M129238 <P>I am trying to submit the spark-sql scala code with yarn-cluster mode and got OOM exception in driver . </P><P><STRONG>command used :</STRONG></P><P>spark-submit --class Test.App --verbose --master yarn-cluster --num-executors 2 --driver-memory 5000m --executor-memory 5000m --executor-cores 2 --driver-cores 2 --conf spark.yarn.driver.memoryOverhead=1024 --conf spark.driver.maxResultSize=5g --driver-java-options "-XX:MaxPermSize=1000m" --conf spark.yarn.jar=hdfs://hdfspath/oozie/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar --jars hdfs://hdfspath/oozie/datanucleus-api-jdo-3.2.6.jar,hdfs://hdfs//oozie/datanucleus-core-3.2.10.jar,hdfs://hdfspath/datanucleus-rdbms-3.2.9.jar,hdfs://hdfs/oozie/mysql-connector-java.jar,hdfs://hdfspath/share/lib/hive/tez-api-0.7.0.2.3.4.0-3485.jar,hdfs://hdfspath/share/lib/hive/tez-dag-0.7.0.2.3.4.0-3485.jar --conf spark.driver.extraJavaOptions="-XX:MaxPermSize=1120m",hive.metastore.uris=thrift://testip:9083,hive.server2.thrift.http.port=10001,hive.server2.thrift.port=10000 --driver-java-options "-Djavax.jdo.option.ConnectionURL=jdbc:mysql://testip/hive?createDatabaseIfNotExist=true -Dhive.metastore.uris=thrift://testip:9083 -Dhive.server2.thrift.port=10000 -Dhive.metastore.warehouse.dir=/apps/hive/warehouse" --files hdfs://hdfspath/oozie/hive-tez-site.xml --driver-class-path hive-tez-site.xml hdfs://hdfspath/oozie/Test.jar 2016-04-11</P><P><STRONG>Error details :</STRONG></P><PRE>16/04/12 18:25:19 INFO hive.HiveContext: default warehouse location is /apps/hive/warehouse 16/04/12 18:25:19 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. 16/04/12 18:25:19 INFO client.ClientWrapper: Inspected Hadoop version: 2.2.0 16/04/12 18:25:19 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.2.0 16/04/12 18:25:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/04/12 18:25:20 INFO hive.metastore: Trying to connect to metastore with URI thrift://test.com:9083 16/04/12 18:25:20 INFO hive.metastore: Connected to metastore. Exception in thread "Driver" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Driver" 16/04/12 18:25:24 INFO spark.SparkContext: Invoking stop() from shutdown hook 16/04/12 18:25:24 INFO history.YarnHistoryService: Application end event: SparkListenerApplicationEnd(1460499924685) </PRE><P>The same code works fine in spark-submit with yarn-client mode . </P><P>I am getting this exception while using HiveContext only. </P><P>Thanks in advance. </P> Wed, 13 Apr 2016 05:44:13 GMT nelsonarasu 2016-04-13T05:44:13Z