I think this is an area where the mismatch between HBase's data model and the traditional relational model cause some weirdness - from what I understand, HBase doesn't have a idea of an arbitrary order of columns in a table - I think Impala is not preserving the creation order when reloading the table.   There are some known issues in this area, maybe we could do better: https://issues.apache.org/jira/browse/IMPALA-886 ... View more

Are there any NULLs in idvar? If so, you could be getting tripped up by the interaction between NOT IN and NULL values. One interesting quirk of SQL is that in some cases IN and NOT IN can both be false for the same row and subquery. E.g. I can recreate a similar scenario if the only value in the subquery is a NULL.   [localhost:21000] > select count(distinct int_col) from functional.alltypestiny; +-------------------------+ | count(distinct int_col) | +-------------------------+ | 2 | +-------------------------+ [localhost:21000] > select count(distinct int_col) from functional.alltypestiny t1 where int_col in (select distinct int_col from functional.alltypesagg where int_col is null); +-------------------------+ | count(distinct int_col) | +-------------------------+ | 0 | +-------------------------+ [localhost:21000] > select count(distinct int_col) from functional.alltypestiny t1 where int_col not in (select distinct int_col from functional.alltypesagg where int_col is null); +-------------------------+ | count(distinct int_col) | +-------------------------+ | 0 | +-------------------------+ I suspect it might be easier to understand if you use a NOT EXISTS. It is almost equivalent to NOT IN but the handling of NULL values is more intuitive.   [localhost:21000] > select count(distinct int_col) from functional.alltypestiny t1 where not exists(select distinct int_col from functional.alltypesagg t2 where int_col is null and t1.int_col = t2.int_col); +-------------------------+ | count(distinct int_col) | +-------------------------+ | 2 | +-------------------------+ [localhost:21000] > select count(distinct int_col) from functional.alltypestiny t1 where exists(select distinct int_col from functional.alltypesagg t2 where int_col is null and t1.int_col = t2.int_col); +-------------------------+ | count(distinct int_col) | +-------------------------+ | 0 | +-------------------------+ ... View more

It looks like it's referencing a Java class "org.apache.hive.jdbc.HiveDriver" that's not available on the Impala JVM's classpath. In Hive I assume that that class is deliberately or accidentally on the classpath of the java tasks spawned for the Hive job. Impala doesn't have that class on it's classpath.   It seems pretty weird for a UDF to depend on a use a database driver, so my first thought is to modify the UDF to removed the need for the dependency. Otherwise, generally, if you want to use Java classes from your UDF they would need to be built into the UDF JAR. ... View more