Impala does not show data accurately after cluster restart

Question

We noticed for few of the hive tables, during the  maintenance window, just after the cluster restart, impala does not show the data accurately  but it shows properly in hive. Once we perform invalidate metadata and refresh on the underlying table, it starts working fine. It was okay to use the work around but this is happening every month consistently after the cluster restart and so we needed to find the root cause and solution. Surprisingly issue is reported for only two tables, one of them is view. These are external hive table with parquet format.  How to investigate the issue any further?

robbiez · Answer

This issue sounds like Impala had stale metadata. But after Impala starts up, it doesn't have any cached metadata so it will load metadata from HMS and HDFS NN immediately (if load_catalog_in_background is true) or when we run a query. So the metadata should be up-to-date. Is it possible that there is some special workflow in your maintenance routine so the tables are updated after Impala loads the metadata? You can check the timestamp of the files in the tables.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.