Thank you for feedback. 1. Increasing shuffle.partitions led to error : Total size of serialized results of 153680 tasks (1024.0 MB) is bigger than spark.driver.maxResultSize (1024.0 MB) 2. Using CLUSTER BY in the select reduced data shuffling from 250 GB to 1 GB and execution time was reduced from 13min to 5min. So it is a good gain. However, I was expecting that I could persist this bucketing to have a minimum shuffling, but it seems that it is not possible, Hive and Spark are not really compatible on this topic. ... View more

Thanks for feedback. For broadcast variables, it is not so much applicable in my case as I have big tables. Concerning filterpushdown, it has not brought results, on the contrary, execution time took longer. ... View more