Hi @Yurii thanks for the bug report - we'll look into it. What version of Impala did you see this on?
One thing worth trying is changing the PARQUET_ARRAY_RESOLUTION query option to THREE_LEVEL
We have sometimes seen similar symptoms with parquet nested types because of some ambiguity in encodings, e.g. https://issues.apache.org/jira/browse/IMPALA-4725. The original version of Impala's nested types defaulted to detecting an older representation of arrays first and we had to keep that behaviour in Impala 2.x/CDH5.x for backwards compatibility. In Impala 3.0/CDH6.0 we're defaulting to the standard array resolution method.
The attached Parquet file seems corrupt: the definition level of nested NULL values are incorrectly filled.
What tool was used to create the file?
For more information about the problem with file see https://issues.apache.org/jira/browse/IMPALA-7471?focusedCommentId=16650568&page=com.atlassian.jira....
Can you give more information about the way it was created?
If we could reproduce it, then a ticket could be created for the writer compenent.
I was created such table using Hive, but I was unable to reproduce the issue:
CREATE TABLE test_users_flat (
insert into test_users_flat values
(1, "user A", "device A1", "phone A1"),
(1, "user A", "device A2", "NULL"),
(2, "user B", "device B1", "NULL"),
(2, "user B", "device B2", "phone B2"),
(2, "user B", "device B3", "phone B3");
INSERT INTO test_users
SELECT id, name, collect_list(named_struct('id', device_id, 'device_info', named_struct('model', device_model)))
FROM test_users_flat GROUP BY id, name;
The resulting table could be read with Impala without problem:
SELECT u.name, d.device_info.model as model
FROM test_users u,
1 user A phone A1
2 user A NULL
3 user B NULL
4 user B phone B2
5 user B phone B3
Can you provide more information like CDH version and the exact steps used to fill the table?
I have checked the writer in the file's metadata, and it is Parquet.Net version 220.127.116.118.
So it seems that this is not an Impala reader issue, but a Parquet.Net writer issue. The definition levels of NULLs in collections are wrong (according to Parquet spec).
This issue it causes is that if the first column read is the collection with NULL in the row, then the 0 def level is interpreted as "the whole row is NULL". If there is another (non NULL) column read first, then its def will be used to determine parents's NULLness, so it will not be NULL. This is why adding 'id' leads to returning the expected results. I would not consider this a bug, rather an optimisation (checking every columns's def level could affect performance).
Parquet.Net is not part of CDH and is not an Apachee project at the moment. I am not familiar with the project, so I do not know whether this is a known issue or not. My advice is to contact the maintainer mentioned at https://github.com/elastacloud/parquet-dotnet