I am trying to submit the spark-sql scala code with yarn-cluster mode and got OOM exception in driver . command used : spark-submit --class Test.App --verbose --master yarn-cluster --num-executors 2 --driver-memory 5000m --executor-memory 5000m --executor-cores 2 --driver-cores 2 --conf spark.yarn.driver.memoryOverhead=1024 --conf spark.driver.maxResultSize=5g --driver-java-options "-XX:MaxPermSize=1000m" --conf spark.yarn.jar=hdfs://hdfspath/oozie/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar --jars hdfs://hdfspath/oozie/datanucleus-api-jdo-3.2.6.jar,hdfs://hdfs//oozie/datanucleus-core-3.2.10.jar,hdfs://hdfspath/datanucleus-rdbms-3.2.9.jar,hdfs://hdfs/oozie/mysql-connector-java.jar,hdfs://hdfspath/share/lib/hive/tez-api-0.7.0.2.3.4.0-3485.jar,hdfs://hdfspath/share/lib/hive/tez-dag-0.7.0.2.3.4.0-3485.jar --conf spark.driver.extraJavaOptions="-XX:MaxPermSize=1120m",hive.metastore.uris=thrift://testip:9083,hive.server2.thrift.http.port=10001,hive.server2.thrift.port=10000 --driver-java-options "-Djavax.jdo.option.ConnectionURL=jdbc:mysql://testip/hive?createDatabaseIfNotExist=true -Dhive.metastore.uris=thrift://testip:9083 -Dhive.server2.thrift.port=10000 -Dhive.metastore.warehouse.dir=/apps/hive/warehouse" --files hdfs://hdfspath/oozie/hive-tez-site.xml --driver-class-path hive-tez-site.xml hdfs://hdfspath/oozie/Test.jar 2016-04-11 Error details : 16/04/12 18:25:19 INFO hive.HiveContext: default warehouse location is /apps/hive/warehouse
16/04/12 18:25:19 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
16/04/12 18:25:19 INFO client.ClientWrapper: Inspected Hadoop version: 2.2.0
16/04/12 18:25:19 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.2.0
16/04/12 18:25:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/04/12 18:25:20 INFO hive.metastore: Trying to connect to metastore with URI thrift://test.com:9083
16/04/12 18:25:20 INFO hive.metastore: Connected to metastore.
Exception in thread "Driver"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Driver"
16/04/12 18:25:24 INFO spark.SparkContext: Invoking stop() from shutdown hook
16/04/12 18:25:24 INFO history.YarnHistoryService: Application end event: SparkListenerApplicationEnd(1460499924685)
The same code works fine in spark-submit with yarn-client mode . I am getting this exception while using HiveContext only. Thanks in advance.
... View more
