Hi, I am currently using Spark streaming to write to an external hive table every 30 mins.
rdd.toDF().write.partitionBy("dt").options(options).format("orc").mode(SaveMode.Append).saveAsTable("table_name")
The issue with this is it creates lots of small files in HDFS, like so part-00000
part-00000_copy_1 My table was created with transactions enabled, and I have enabled ACID transactions on the Hive instance however, I can't see any compactions running nor do any get created when I force compaction with ALTER TABLE command. I would expect compaction to run and merge these files as they are very small 200 KB's in size. Any idea's or help greatly appreciated
... View more
