Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎08-14-2018

Impala bug with nested arrays of structures where some of the fields contains null

Hi All,
 
We found a case where Impala returns incorrect values from simple query. Our data contains nested array of structures and structures contains other structures.
We generated minimal sample data allowing to reproduce the issue.
 
SQL to create a table:
CREATE TABLE plat_test.test_users (
  id INT,
  name STRING,   
  devices ARRAY<
    STRUCT<
      id:STRING,
      device_info:STRUCT<
        model:STRING
      >
    >
  >
)
STORED AS PARQUET
 
Please put attached parquet file to the location of the table and refresh the table.
In sample data we have 2 users, one with 2 devices, second one with 3. Some of the devices.device_info.model fields are NULL.
 
When I issue a query:
SELECT u.name, d.device_info.model as model
FROM test_users u,
u.devices d;
 
I'm expecting to get 5 records in results, but getting only one1.png
If I change query to:
SELECT u.name, d.device_info.model as model
FROM test_users u
LEFT OUTER JOIN u.devices d;
 
I'm getting two records in the results, but still not as it should be:
 
2.png
We found some workaround to this problem. If we add to the result columns device.id we will get all records from parquet file:
SELECT u.name, d.id, d.device_info.model as model
FROM test_users u
, u.devices d
 
And result is
 
3.png
 
But we can't rely on this workaround, because we don't need device.id in all queries and Impala optimizes it, and as a result we are getting unpredicted results.
 
I tested Hive query on this table and it returns expected results:
SELECT u.name, d.device_info.model
FROM test_users u
lateral view outer inline (u.devices) d;
 
results:
4.png
Please advice if it's a problem in Impala engine or we did some mistake in our query.
 
 
Best regards,
Come2Play team.
Cloudera Employee
Posts: 347
Registered: ‎07-29-2015

Re: Impala bug with nested arrays of structures where some of the fields contains null

Hi @Yurii thanks for the bug report - we'll look into it. What version of Impala did you see this on?

 

One thing worth trying is changing the PARQUET_ARRAY_RESOLUTION query option to THREE_LEVEL

https://www.cloudera.com/documentation/enterprise/latest/topics/impala_parquet_array_resolution.html

 

We have sometimes seen similar symptoms with parquet nested types because of some ambiguity in encodings, e.g. https://issues.apache.org/jira/browse/IMPALA-4725. The original version of Impala's nested types defaulted to detecting an older representation of arrays first and we had to keep that behaviour in Impala 2.x/CDH5.x for backwards compatibility. In Impala 3.0/CDH6.0 we're defaulting to the standard array resolution method.

Announcements