Hello @Shelton , thanks for providing the informative details on Hive and Impala but that's not helping with the question I have. I understand that when the cluster restart happens, till the catalogue store is fully refreshed, impala queries would run slow. My question is not about slow running impala queries after restart. The question is very different and I will try to explain it again. We work on a very big CDH cluster with numerous amount of users and applications running on it. We have monthly maintenance schedule where some patch like RHEL, CDH etc is applied to all the hosts and then system is restarted. Its done in off peak hours ( late night thru early morning. ) Next day morning users of "certain" tables start complaining they are missing data in table as they are "impala" users. They can see few but some are missing. The table has year wise partition so let us say they say they can see 2021, 2020 and 2017 but 2018 and 2019 is missing. To troubleshoot, we go to beeline prompt and run same query in hive and we can see every thing. We come back to Impala and do invalidate metadata and refresh and its start working for users too. who are checking thru Impala and they think that I am a magician who can do some magic to bring their missing data 🙂 This had been happening since last 6 months and rather than become a magician every time cluster restart, I needed to find the root cause why this is happening only for certain tables. when the user is trying to query and if the meta store is not refreshed, I expect Impala to take its time on first hit and refresh it and then show every thing accurate. Showing inaccurate result or incomplete result was not part of the deal with Impala. If anyone else has faced similar issue or know what is going on here, please chime in.
... View more
