question Re: ERROR: Parquet file should not be split into multiple hdfs-blocks in Archives of Support Questions (Read Only)

ERROR: Parquet file should not be split into multiple hdfs-blocks

ygnhzeus — Fri, 16 Sep 2022 09:00:50 GMT

Hello,

I'm trying to use parquet fileformat and it works fine if I write data using Impala and read it in Hive. However, if I insert data to that table via Hive and read it using Impala, Impala will throw errors like:

ERRORS:

Backend 2: Parquet file should not be split into multiple hdfs-blocks

...

It seems that this error is not a fatal one and Impala is able to get the query results, what might be the cause and how to avoid this Error?

Thanks!

Re: ERROR: Parquet file should not be split into multiple hdfs-blocks

Harsh J — Sun, 20 Jul 2014 15:20:06 GMT

How large are your Parquet input files?

If you are copying your files around, have you ensured following the block size preservation method mentioned at http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_parquet.html?

Within Hive, you can perhaps "set dfs.blocksize=1g;" before issuing the queries to create the files.

Re: ERROR: Parquet file should not be split into multiple hdfs-blocks

TomasTF — Thu, 30 Oct 2014 10:06:53 GMT

Had the same issue, created a partitioned table stored as parquet in Hive, and loaded with data.

Then whe running the query in Impala got the same error message.

I tried these settings in Hive before running the insert, but the files produced are greater than the HDFS block size (128MB)

SET parquet.block.size=128000000;
SET dfs.blocksize=128000000;

Can anybody give an advice?

Tomas

Re: ERROR: Parquet file should not be split into multiple hdfs-blocks

TomasTF — Fri, 07 Nov 2014 14:08:45 GMT

I solved it with increasing the block size to the largest possible value of the partition, so when one partition is always less than 800MB I set the block size for this table to 1GB, and the warnings do not appear any more.

Re: ERROR: Parquet file should not be split into multiple hdfs-blocks

BrockOwen — Thu, 18 Dec 2014 21:22:13 GMT

How can this be done when writing data from a Pig script?

Re: ERROR: Parquet file should not be split into multiple hdfs-blocks

James K — Mon, 23 Nov 2015 20:59:16 GMT

I'm running into something similar. I'm on 5.4.2 building tables with have then analyzing with Impala and I get the same warnings, although the queries execute ok.

Can you please share with me what you scripted to make "when one partition is always less than 800MB I set the block size for this table to 1GB" as you mention in your post?