I am reading lot of csv files s3 via Spark and writing into a hive table as orc. While writing, it is writing lot of small files. I need to merge all these files, i tried setting the property sqlContext.sql("set hive.merge.sparkfiles=true"). But this has no impact. Given below the code. Please help. sqlContext.sql("set hive.merge.sparkfiles=true") sqlContext.sql("set hive.merge.smallfiles.avgsize=128000000") sqlContext.sql("set hive.merge.size.per.task=128000000") new_df.registerTempTable("new_df") sqlContext.sql("INSERT OVERWRITE TABLE db.tablename " + "PARTITION(`week`='W_1617')" + "SELECT col1, col2 FROM new_df" ... View more