Hi @tusharkathpal ! Thanks for the detailed explanation, really appreciate it! In my case, all tables were created beforehand, so all their static metadata should be already cached. However, there are commands emitted from clients to create partitions in Impala tables from time to time (every hour) and also refresh commands are periodically issued on those new partitions (every minute) to make parquet files inside them available to be queried in Impala. I can confirm that only a handful of tables were being ingested to during the HDFS switchovers. Probably the partition creation on "impala_table" or a refresh command on one of its partitions triggered a fetch of metadata from catalog server, which would explain why it happened only for "impala_table". About hive metatool command, it is listing the correct HDFS locations. I don't think it applies in my case, because HDFS is already deployed with the final nameservice in the config before hadoop starts up (i.e., there is no upgrade from non-HA to HA setup involved). About automatic invalidation of metadata, I will consider it for future Impala upgrades. It would help by handling the metadata change on "alter table add partition" command. However, I would need to change part of the ingestion pipeline due to this use case of adding files directly on the filesystem not supported.
... View more
