Could be a data skew issue. Checkout if any partition has huge chunk of the data compared to the rest. https://github.com/adnanalvee/spark-assist/blob/master/spark-assist.scala From the link above, copy the function "partitionStats" and pass in your data as a dataframe. It will show the maximum, minimum and average amount of data across your partitions like below. +------+-----+------------------+ |MAX |MIN |AVERAGE | +------+-----+------------------+ |135695|87694|100338.61149653122| +------+-----+------------------+ ... View more

Why are you using 10g of driver memory? What is the size of your dataset and how many partitions does it have? I would suggest using the config below: --executor-memory 32G \ --num-executors 20 \ --driver-memory 4g \ --executor-cores 3 \ --conf spark.driver.maxResultSize=3g \ ... View more

@Marcos Da Silva This should solve the problem as it did for mine. select column1,column2 from table where partition_column in (select max(distinct partition_column) from table)" ... View more

@Joe Widen @Timothy Spann ... View more

NOTES: Tried different no. executors from 10-60 but performance doesn't improve. Saving in Parquet format saves 1 minute but I dont want parquet. ... View more

Thanks a lot! ... View more

If you are running on cluster mode, you need to set the number of executors while submitting the JAR or you can manually enter it in the code. The former way is better spark-submit \ --master yarn-cluster \ --class com.yourCompany.code \ --executor-memory 32G \ --num-executors 5 \ --driver-memory 4g \ --executor-cores 3 \ --queue parsons \ YourJARfile.jar \ If running locally, spark-shell --master yarn --num-executors 6 --driver-memory 5g --executor-memory 7g ... View more

@Dinesh Das Coursera has a popular one. https://www.coursera.org/specializations/scala ... View more