The reason you were seeing HdfsParquetTableWriter::ColumnWriter is that I was testing the bug using the syntax:   CREATE TABLE db.newTable STORED AS PARQUET AS SELECT a.topLevelField, b.priceFromNestedField FROM db.table a LEFT JOIN a.nestedField    This was purely to force the bug to occur - if you just did the SELECT in Hue it would often succeed because it only brings back the first 100 rows - to consistently trigger the crash I had to make Impala read from both Parquet files. No other query was running at the time.   Anyway, as Chris says, the bug appears to be fixed in 5.14.2. The job which originally consistently triggered the crash has now been running unchanged over the same source data for 20 hours without hitch.   Thanks for your help Matt     ... View more

I've finally solved using the executeUpdate method:   // invalidate metadata and rebuild index on Impala try { Statement stmt = impalaConn.createStatement(); try { String query = "INVALIDATE METADATA;"; int result = stmt.executeUpdate(query); while (resultSet.next()) { // do something } } finally { stmt.close(); } } catch(SQLException ex) { while (ex != null) { ex.printStackTrace(); ex = ex.getNextException(); } System.exit(1); } Thanks for help! ... View more

Are there any NULLs in idvar? If so, you could be getting tripped up by the interaction between NOT IN and NULL values. One interesting quirk of SQL is that in some cases IN and NOT IN can both be false for the same row and subquery. E.g. I can recreate a similar scenario if the only value in the subquery is a NULL.   [localhost:21000] > select count(distinct int_col) from functional.alltypestiny; +-------------------------+ | count(distinct int_col) | +-------------------------+ | 2 | +-------------------------+ [localhost:21000] > select count(distinct int_col) from functional.alltypestiny t1 where int_col in (select distinct int_col from functional.alltypesagg where int_col is null); +-------------------------+ | count(distinct int_col) | +-------------------------+ | 0 | +-------------------------+ [localhost:21000] > select count(distinct int_col) from functional.alltypestiny t1 where int_col not in (select distinct int_col from functional.alltypesagg where int_col is null); +-------------------------+ | count(distinct int_col) | +-------------------------+ | 0 | +-------------------------+ I suspect it might be easier to understand if you use a NOT EXISTS. It is almost equivalent to NOT IN but the handling of NULL values is more intuitive.   [localhost:21000] > select count(distinct int_col) from functional.alltypestiny t1 where not exists(select distinct int_col from functional.alltypesagg t2 where int_col is null and t1.int_col = t2.int_col); +-------------------------+ | count(distinct int_col) | +-------------------------+ | 2 | +-------------------------+ [localhost:21000] > select count(distinct int_col) from functional.alltypestiny t1 where exists(select distinct int_col from functional.alltypesagg t2 where int_col is null and t1.int_col = t2.int_col); +-------------------------+ | count(distinct int_col) | +-------------------------+ | 0 | +-------------------------+ ... View more