question Re: Possibility Split Parquet file in Support Questions

Possibility Split Parquet file

HadoopHero — Tue, 21 Apr 2026 06:46:30 GMT

Hello everyone,

my team using TEZ, in particular Hive, has noticed that during an insert with a very simple select a single parquet file of 1.5 gb per partition is generated in the output table.

To try to remedy the problem, a number of settings were used at the session level but had no effect.

Below are the sets used at the session level:

SET hive.execution.engine=tez;
SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.optimise.sort.dynamic.partition.threshold=0;
--SET tez.grouping.max-size=268435456;
--SET hive.exec.reducers.bytes.per.reducer=536870912;
--SET tez.grouping.split-count=18;
SET hive.vectorized.execution.reduce.enabled = true;
SET hive.vectorized.execution.reduce.groupby.enabled = true;
--SET hive.tez.auto.reducer.parallelism=false;
--SET mapred.reduce.tasks=12;
--SET hive.tez.partition.size=104857600;
--SET hive.tez.partition.num=10;
SET hive.parquet.output.block.size=104857600;

I would like to ask if there is a way to always have parquet type files but broken up into smaller files as shown in the image below

We cannot understand what the cause might be.
Files structured in this way do not guarantee sufficient parallelism for other jobs present (such as sqoop)

Re: Possibility Split Parquet file

mszurap — Wed, 08 Nov 2023 09:27:41 GMT

Hi @HadoopHero ,

For Hive, if there is a single reduce task to write the output data it will not break it up the output file into smaller files, that's expected and cannot be configured to behave in a different way.

With DISTRIBUTE BY you should be able to achieve to have multiple reducers (if you have a column by which you can "split" your data reasonably into smaller subsets), see

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+SortBy

Best regards

Miklos

Re: Possibility Split Parquet file

HadoopHero — Thu, 09 Nov 2023 08:51:05 GMT

Hello Miklos,

unfortunately, what you suggested had no effect. We continue to have the same problem, with creating a single parquet file.

Re: Possibility Split Parquet file

ggangadharan — Thu, 09 Nov 2023 10:56:47 GMT

Hi @HadoopHero ,

If the query involves dynamic partitioning, one potential issue is that 'hive.optimize.sort.dynamic.partition.threshold' may limit the number of open record writers to just one per partition value, resulting in the creation of only one file. To investigate this, could you attempt disabling 'hive.optimize.sort.dynamic.partition.threshold' entirely?

SET hive.optimize.sort.dynamic.partition.threshold=-1;

Note : The problem statement contains a typo in the config name

Re: Possibility Split Parquet file

DianaTorres — Mon, 13 Nov 2023 17:32:20 GMT

@HadoopHero Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

Re: Possibility Split Parquet file

HadoopHero — Mon, 13 Nov 2023 22:13:18 GMT

@DianaTorres I'm sorry but unfortunately the problem still persists even after trying the suggestions in the previous posts

Re: Possibility Split Parquet file

DianaTorres — Mon, 13 Nov 2023 23:36:11 GMT

@Shmoo @cravani Do you have any insights here? Thanks!

Re: Possibility Split Parquet file

cravani — Tue, 21 Nov 2023 04:24:36 GMT

@HadoopHero Answer would vary based on query that you are running, assuming you have simple "Insert select */cols from Table" it is likely mapper only job and you may want to try tuning below.

set tez.grouping.min-size=134217728; -- 128 MB min split set tez.grouping.max-size=1073741824; -- 1 GB max split

Try setting min-size and max-size to same value. I would not go below 128M.

Re: Possibility Split Parquet file

DianaTorres — Fri, 24 Nov 2023 17:31:35 GMT

@HadoopHero Has the reply helped resolve your issue? Thanks.