Skip to main content
npr20202
npr20202
Silver ⭐️⭐️
August 26, 2021
Question

Impala does not show data accurately after cluster restart

We noticed for few of the hive tables, during the  maintenance window, just after the cluster restart, impala does not show the data accurately  but it shows properly in hive. Once we perform invalidate metadata and refresh on the underlying table, it starts working fine. It was okay to use the work around but this is happening every month consistently after the cluster restart and so we needed to find the root cause and solution. Surprisingly issue is reported for only two tables, one of them is view. These are external hive table with parquet format.  How to investigate the issue any further? 

9 replies

robbiez
Gold ⭐️
robbiez
Gold ⭐️
August 26, 2021

This issue sounds like Impala had stale metadata. But after Impala starts up, it doesn't have any cached metadata so it will load metadata from HMS and HDFS NN immediately (if load_catalog_in_background is true) or when we run a query. So the metadata should be up-to-date. Is it possible that there is some special workflow in your maintenance routine so the tables are updated after Impala loads the metadata? You can check the timestamp of the files in the tables.

Translate
npr20202
npr20202Author
Silver ⭐️⭐️
August 29, 2021

Thanks for your reply but during maintenance windows only CDH patching or RHEL patching kind of activities are done and no script runs on the application tables. so as you suggested after the restart, Impala should not have any cached metadata. However the issue is happening consistently every month after the restart. Any suggestion what to check and investigate when next time cluster is restarted and we notice the issue. 

Translate
npr20202
npr20202Author
Silver ⭐️⭐️
September 19, 2021

This time we did show table stats and show column stats on the table before the issue ( just after the restart and before running invalidate metadata) and then after the restart and we do not notice any difference in output but problem reoccurred this time also.  To understand the sequence of events :

(1) Admin team does patching and restart the cluster

(2) application team run a distinct year query and sees out of 31 only 27 rows are present and 4 of the partitions are missing. 

(3) as a support team, I run show stats which shows all 31 partitions and number of rows in partitions is correct and size also.

(4) we run invalidate metadata and refresh

(5) support team takes show stats which has same result

(6) app team runs they query again and they able to see all 31 partitions.

 

we will work with vendor to get further help. 

Translate
Terms & ConditionsAccessibility statement