question Merge small files in pyspark for Hive table in Support Questions

Merge small files in pyspark for Hive table

drgenious — Tue, 14 Dec 2021 07:21:30 GMT

I have an ETL flow which transfers data from a hive table to another through pyspark. The tables are partitioned. Although I see that in the partition's path in HDFS there are small parquet files. I want to ask:
1)How can I merge these files?
2)Is there any max size or recommended size for hive partitions?

Re: Merge small files in pyspark for Hive table

sree_nv — Fri, 07 Jan 2022 13:45:54 GMT

@drgenious

1)If those small files are delta files, you can run compaction manually to merge all of them

more details on manual compaction described in the link below:
https://docs.cloudera.com/runtime/7.2.10/managing-hive/topics/hive_initiate_hive_compaction.html

2)Current Hive versions with RDBMS metastore backend should be able to handle 10 000+ partitions.

Thanks,

Sree

Re: Merge small files in pyspark for Hive table

asish — Sat, 08 Jan 2022 12:40:56 GMT

@drgenious

You can use set hive.merge.tezfiles=true; to fix merge file issue