How much memory do you have? How much is assigned to Spark? Do you have logging on so you can check logs and history UI? Turn off everything else you can. For debugging run through the Spark shell, Zeppelin adds over head and takes a decent amount of YARN resources and RAM. Run on Spark 1.6 / HDP 2.4.2 if you can. Allocate as much memory as possible. Spark is an all memory beast. sparkConf.set("spark.cores.max", "16") // all the cores you can sparkConf.set("spark.serializer", classOf[KryoSerializer].getName) sparkConf.set("spark.sql.tungsten.enabled", "true") sparkConf.set("spark.eventLog.enabled", "true") sparkConf.set("spark.app.id", "YourID") sparkConf.set("spark.io.compression.codec", "snappy") sparkConf.set("spark.rdd.compress", "true")
I like to maximize my resources and performance.
... View more
