whenever i am running either select or insert query on Impala, it takes huge timing. after seeing the log i can see most of the times going in loading the table. i see message like this :
Frontend.java:808] Requesting prioritized load of table(s): comm_tran_data,comm_tran_info
I cant understand when every impalad have catalog with all the metadata then why its trying to load.
I am using Impala version 2.7,2.9 and 2.1. in all the version same problem.
can you suggest the solution for it.?? this problem is coming in PROD env.
Impala doesn't eagerly load all metadata into its cache, it only does so the first time that the table is referenced (or after a REFRESH or INVALIDATE METADATA that affects the table).
Loading metadata can take a long time if your table has a lot of partitions or files or if there is another query that is loading a large table.
any updates on this issue please this is really very irritating. when i see catalog tab i see all the tables loaded there. but when the same table used in query it goes like :
as src where src.r = 1 I1029 15:35:41.023432 21809 Frontend.java:822] Requesting prioritized load of table(s): default.comm_simeod_raw
i just do refresh whenever any partition is added in the table.even if i run the query on old data same problem comes.
not all the times but most of the times, table size is not even 100 gb. there are 1200 partitions in the table. my question is when the table is loaded in catalog which i can see by going to that tab then what does it mean by loading ? and why its doing ?