Hi @RonyA You haven't shared what is your dataset size. Apart from data you need to tune on few spark parameters. spark = (SparkSession
.builder.master("yarn")
.config("spark.executor.cores", "5") # you have mentioned 12
.config("spark.num.executors", "10")
.config("spark.executor.memory", "10G")
.config("spark.executor.memoryOverhead", "2G") # executor memory * 0.1 or 0.2 %
.config("spark.driver.memory", "10G")
.config("spark.driver.memoryOverhead", "2G") # driver memory * 0.1 or 0.2 %
.config("spark.sql.hive.convertMetastoreOrc", "true")
.config("spark.executor.heartbeatInterval", "60s") # default 10s
.config("spark.network.timeout", "600s") # default 120s
.config("spark.driver.maxResultSize", "2g")
.config("spark.driver.cores","4")
.config("spark.executor.extraClassPath", "-Dhdp.version=current")
.config("spark.debug.maxToStringFields", 200)
.config("spark.sql.catalogImplementation", "hive")
.config("spark.memory.fraction", "0.8")
.config("spark.memory.storageFraction", "0.2")
.config("spark.sql.hive.filesourcePartitionFileCacheSize", "0")
.config("spark.yarn.maxAppAttempts", "10")
.appName(app_name)
.enableHiveSupport().getOrCreate()) Apart from above if you are doing any kind of wide operation shuffle is involved. To set shuffle value we will use below calculation: spark.sql.shuffle.partitions = shuffle input size/ hdfs block size for example, shuffle input size is 10GB and hdfs block size is 128 MB then shuffle partitions is 10GB/128MB = 80 partitions. And also check you have enabled dynamic allocation or not. You can open Spark UI --> Select Application --> Go to the Environment page --> find spark.dynamicallocation.enabled property.
... View more
