question Re: Parquet files should not be split into multiple hdfs-blocks problem and strange record count iss in Archives of Support Questions (Read Only)

Parquet files should not be split into multiple hdfs-blocks problem and strange record count issue

msuluhan — Fri, 16 Sep 2022 10:13:12 GMT

Hello

I am trying to import parquet tables from another Cloudera Impala implementation to my Cloudera Impala

--> I am getting parquet tables via sftp

--> I am copying all parquet files into proper impala table directory like /grid1/hive/warehouse/<database>/<importedTable> without any error/warning

--> I am creating required partition structure with alter table <importedTable> add partition (..) without any error/warning

--> I am applying refresh <importedTable> command without any error/warning

--> I could see new partitions in (show partition <importedTable> command) without any error/warning

--> I am applying above procedure for all tables

--> When I tried to access records in the table I got following warning "WARNINGS: Parquet files should not be split into multiple hdfs-blocks"

I am using gzip compression on my tables but imported tables have default settings. So I have another database with gzipped data. Therefore I am copying data from imported table to gzipped table with following command

set compression_codec=gzip without any error/warning

insert into <gzippedTable> partition (<part1=value1, part2=value2) select field1, field3, field4 ...... from <importedTable> where <partitioned column1=value1, partitioned column2=value2) without any error/warning

When I compare record counts for the partition both gzippedtable and imported table, there is a differences like following output

[host03:21000] > select count (*) from importedTable where logdate=20160401;

Query: select count (*) from importedTable where logdate=20160401

+-----------+

| count(*) |

+-----------+

| 101565867 |

+-----------+

WARNINGS: Parquet files should not be split into multiple hdfs-blocks. file=hdfs://host01:8020/grid1/hive/warehouse/<database>/importedTable/partitionedColumn=value1/logdate=20160401/51464233716089fd-295e6694028850a0_1358598818_data.0.parq (1 of 94 similar)

Fetched 1 row(s) in 0.96s

[host03:21000] > select count (*) from gzippedTable where logdate=20160401;

Query: select count (*) from gzippedTable where logdate=20160401

+-----------+

| count(*) |

+-----------+

| 123736525 |

+-----------+

Fetched 1 row(s) in 0.92s

So how can I fix "WARNINGS: Parquet files should not be split into multiple hdfs-blocks" and why I am getting different record counts after applying above procedure.

Is record count differences related with multiple hdfs-blocks warning ?

Thanks

Re: Parquet files should not be split into multiple hdfs-blocks problem and strange record count iss

TomScott — Mon, 18 Apr 2016 09:34:09 GMT

Regarding the "WARNINGS: Parquet files should not be split into multiple hdfs-blocks" issue, what is the HDFS block size set to for the application that is inserting the parquet data into HDFS? If your application is using the default this should be found in hdfs-site.xml under the dfs.blocksize property.

If this is smaller than the size of the parquet files then they will be split into multiple HDFS blocks. This can mean Impala has to read one or more blocks remotely to reassemble a parquet row group. This carries performance impact. This is explained pretty well here: http://ingest.tips/2015/01/31/parquet-row-group-size/

Please note that dfs.blocksize can be set per application so you should not need to modify the global setting. Try raising this to your largest parquet file size when inserting to HDFS.

Re: Parquet files should not be split into multiple hdfs-blocks problem and strange record count iss

msuluhan — Mon, 09 May 2016 15:59:17 GMT

Hello

So sorry for delayed update.

invalidate metadata
invalidate metadata tablename and then
refresh tablename

commands have solved my problem. Source parquet tables and gzipped target tables have same records in their partitions. I am still getting "split into multiple hdfs-blocks problem" warnings but it looks like it does not any impact on my record count issue.

BTW : The link that you provided is very good

Thanks for your response

Re: Parquet files should not be split into multiple hdfs-blocks problem and strange record count iss

hrishi1dypim — Thu, 19 Sep 2019 03:53:44 GMT

Not able to open this link : http://ingest.tips/2015/01/31/parquet-row-group-size/

can you please check and repost it please ?