question Limit number of parquet files when doing an insert/create in impala in Archives of Support Questions (Read Only)

Limit number of parquet files when doing an insert/create in impala

gimp077 — Fri, 16 Sep 2022 12:22:53 GMT

I am doing something like this

create table test2 stored as parquet as select * from t1;

And I would like to make sure that only 2 parquet files are created lets say is this possible somehow.

As know there is no predictable threshold for how many files will be created.

Thanks

Re: Limit number of parquet files when doing an insert/create in impala

mauricio — Wed, 11 Oct 2017 03:05:30 GMT

You can do "set NUM_NODES=1" in your session (before your query), which will cause it to be processed in a single node (just in the coordinator). It will produce 1 file, up to the default max size of parquet files.

You can do "set PARQUET_FILE_SIZE=XX" to fine-tune that max file size up or down until you get it split exactly into 2 files (it will take some trial and error because this is an upper bound - files are actually quite a bit smaller than the limit in my experience).

But beware the docs state NUM_NODES is not for production use, especially on big tables, as it can put a lot of pressure on a single host and crash that impalad.

https://www.cloudera.com/documentation/enterprise/latest/topics/impala_query_options.html

-m

Re: Limit number of parquet files when doing an insert/create in impala

mauricio — Wed, 11 Oct 2017 03:10:31 GMT

Another option I forgot to mention: if your table is partitioned, and your insert query uses dynamic partitioning, it will generate 1 file per partition:

insert into table2 partition(par1,par2) select col1, col2 .. colN, par1, par2 from table1;

... again up to the max parquet file size currently set, so you can play with that max to achieve 2 files per partition.

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/impala_partitioning.html#partition_static_dynamic