I am only dealing with Parquet and Avro luckily, not text. And yes, I was referring to the key partitions in the files.   Sorry for going off topic, but I'm still quite new to Spark and the whole Hadoop ecosystem in general, so I'm still trying to get a feel for everything. To clarify, partitions of a RDDs/Dataframes are different than the key based partitions of the files? I had always thought they were the same. ... View more

I should have mentioned it in the first post, but I need to maintain existing partitions as they are, so I need to compact files within partitions. ... View more