Hey @bkosaraju , thanks for sharing your thoughts. I had this as one of the alternatives but looking is there any other way in Spark SQL like transaction level control (like commit or rollback) if a data frame is not created or for any exception. Thanks you. ... View more

Hi @Vijayalakshmi Ekambaram, Spark default nature is to perform 200 partitions when doing aggregations , which is defined by the conf variable "spark.sql.shuffle.partitions" (default value is 200). This is the reason you will find lot of small files in the hive URI after each insert into hive table using Spark. you can use the below "coalesce" statement to avoid writing many files into hive. (Coalesce won't perform full shuffle where as repartition does a full shuffle of data across network) new_df.coalesce(2).write.mode("append").partitionBy("week").insertInto(db.tablename) Note: The higher number is introduced to increase the amount of parallelism in Spark, which is useful when we have huge workload. ... View more