Yes increase driver and executor memory. http://spark.apache.org/docs/latest/configuration.html Also specify more executors. in spark submit --master yarn --deploy-mode cluster In my code I like to: val sparkConf = new SparkConf().setAppName("Links")
sparkConf.set("spark.cores.max", "32")
sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)
sparkConf.set("spark.sql.tungsten.enabled", "true")
sparkConf.set("spark.eventLog.enabled", "true")
sparkConf.set("spark.app.id", "MyApp")
sparkConf.set("spark.io.compression.codec", "snappy")
sparkConf.set("spark.rdd.compress", "false")
sparkConf.set("spark.suffle.compress", "true")
See http://spark.apache.org/docs/latest/submitting-applications.html --executor-memory 32G \
--num-executors 50 \ Up these spark.driver.cores 32 Number of cores to use for the driver process, only in cluster mode. spark.driver.maxResultSize 1g Limit of total size of serialized results of all partitions for each Spark action (e.g. collect). Should be at least 1M, or 0 for unlimited. Jobs will be aborted if the total size is above this limit. Having a high limit may cause out-of-memory errors in driver (depends on spark.driver.memory and memory overhead of objects in JVM). Setting a proper limit can protect the driver from out-of-memory errors. spark.driver.memory 32g Amount of memory to use for the driver process, i.e. where SparkContext is initialized. (e.g. 1g , 2g ).
Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-memory command line option or in your default properties file. spark.executor.memory 32g Amount of memory to use per executor process (e.g. 2g , 8g ).
See http://spark.apache.org/docs/latest/running-on-yarn.html for running on YARN --driver-memory 32g \
--executor-memory 32g \ --executor-cores 1 \ Use as much memory as you can for optimal Spark performance. spark.yarn.am.memory 16G Amount of memory to use for the YARN Application Master in client mode, in the same format as JVM memory strings (e.g. 512m , 2g ). In cluster mode, use spark.driver.memory instead. Use lower-case suffixes, e.g. k , m , g , t , and p , for kibi-, mebi-, gibi-, tebi-, and pebibytes, respectively. spark.driver.memory 16g Amount of memory to use for the driver process, i.e. where SparkContext is initialized. (e.g. 1g , 2g ).
Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-memory command line option or in your default properties file. spark.driver.cores 50 Number of cores used by the driver in YARN cluster mode. Since the driver is run in the same JVM as the YARN Application Master in cluster mode, this also controls the cores used by the YARN Application Master. In client mode, use spark.yarn.am.cores to control the number of cores used by the YARN Application Master instead.
Up things where you can and run in YARN cluster mode. Good Reference: http://www.slideshare.net/pdx_spark/performance-in-spark-20-pdx-spark-meetup-81816 http://www.slideshare.net/JenAman/rearchitecting-spark-for-performance-understandability-63065166 See the Hortonworks Spark Tuning Guide http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_spark-component-guide/content/ch_tuning-spark.html
... View more
