Skip to main content
npr20202
npr20202
Silver ⭐️⭐️
August 26, 2021
Question

Impala does not show data accurately after cluster restart

We noticed for few of the hive tables, during the  maintenance window, just after the cluster restart, impala does not show the data accurately  but it shows properly in hive. Once we perform invalidate metadata and refresh on the underlying table, it starts working fine. It was okay to use the work around but this is happening every month consistently after the cluster restart and so we needed to find the root cause and solution. Surprisingly issue is reported for only two tables, one of them is view. These are external hive table with parquet format.  How to investigate the issue any further? 

9 replies

robbiez
Gold ⭐️
robbiez
Gold ⭐️
August 26, 2021

This issue sounds like Impala had stale metadata. But after Impala starts up, it doesn't have any cached metadata so it will load metadata from HMS and HDFS NN immediately (if load_catalog_in_background is true) or when we run a query. So the metadata should be up-to-date. Is it possible that there is some special workflow in your maintenance routine so the tables are updated after Impala loads the metadata? You can check the timestamp of the files in the tables.

Translate
Shelton
Shelton
Gold ⭐️⭐️⭐️
August 26, 2021

@npr20202 

To return accurate query results, Impala needs to keep the metadata current for the databases and tables queried. Therefore, if some other entity modifies information used by Impala in the metastore, the information cached by Impala must be updated via INVALIDATE METADATA or REFRESH.

Difference between INVALIDATE METADATA and REFRESH

INVALIDATE METADATA

Spoiler
is an asynchronous operation that simply discards the loaded metadata from the catalog and coordinator caches. After that operation, the catalog and all the Impala coordinators only know about the existence of databases and tables and nothing more. Metadata loading for tables is triggered by any subsequent queries.

REFRESH

Spoiler
Just reloads the metadata synchronously. REFRESH is more lightweight than doing a full metadata load after a table has been invalidated. REFRESH cannot detect changes in block locations triggered by operations like HDFS balancer, hence causing remote reads during query execution with negative performance implications.

Syntax

INVALIDATE METADATA [[db_name.]table_name]

You can run it in the HUE or impala-shell

i.e

INVALIDATE METADATA product.customer

By default, the cached metadata for all tables is flushed. If you specify a table name, only the metadata for that one table is flushed. Even for a single table.

INVALIDATE METADATA is more expensive than REFRESH, so prefer REFRESH in the common case where you add new data files for an existing table.

Translate
Terms & ConditionsAccessibility statement