We had a Hive metastore crash and the metastore server was back online in few minutes. Post the crash and restart, we noticed that Impala threw 'TableLoadingException: Failed to load metadata for table' for certain Table(s) despite returning proper results just before the crash as well. Running an invalidate metadata on the table eventually fixed the problem since it loaded the metadata again. However it was not a new table and the behaviour is puzzling.
Following are the sequence of events :
7:54 AM UTC - Data ingestion for the table through Hive. 7:54 AM UTC - Table refreshed in Impala. 7:55 AM UTC - Invalidate metadata at 9:27 AM UTC - Latest successful query on the table. Metastore crashed at 9:35 AM UTC and restarted automatically by Cloudera. There are no logs or trace or dump for the crash.
first failed query at: 9:36 AM UTC
The Impalad continued to return the same exception 'TableLoadingException: Failed to load metadata for table' until an Invalidate metadata command was fired against the table.
Following are the questions :
1. The table was invalidated a couple of hours before and I presume the impala daemons should have the metadata about the table that it need not request any reload of metadata from catalog daemon until forcefully done through refresh/Invalidate metadata. 2. Besides refresh and Invalidate metadata, does the Impalad requests reload of metadata for select queries to catalog server. 3. Is this a bug or an expected behaviour? If so what is unique in this set of actions?