Requesting prioritized load of table(s):

[#1]

Hello,

whenever i am running either select or insert query on Impala, it takes huge timing. after seeing the log i can see most of the times going in loading the table. i see message like this :

Frontend.java:808] Requesting prioritized load of table(s): comm_tran_data,comm_tran_info

I cant understand when every impalad have catalog with all the metadata then why its trying to load.

I am using Impala version 2.7,2.9 and 2.1. in all the version same problem.

can you suggest the solution for it.?? this problem is coming in PROD env.

[#2]

Impala doesn't eagerly load all metadata into its cache, it only does so the first time that the table is referenced (or after a REFRESH or INVALIDATE METADATA that affects the table).

Loading metadata can take a long time if your table has a lot of partitions or files or if there is another query that is loading a large table.

[#3]

After starting impala,I do invalidate metadata on all the impalad nodes so
should I still get this problem ?

[#4]

If you run INVALIDATE METADATA often, you will get this issue more often as well. You should avoid running INVALIDATE METADATA, and try to use REFRESH when practical, or only INVALIDATE METADATA a particular table you need.

Do not try to refresh whole impala metadata too often!

[#5]

On startup I do invalidate metadata. After that whenever any file is loaded
I run invalidate metadata for the table. I don't run refresh table because
it takes longer time than invalidate metadata table.

[#6]

any updates on this issue please this is really very irritating. when i see catalog tab i see all the tables loaded there. but when the same table used in query it goes like :

as src where src.r = 1
I1029 15:35:41.023432 21809 Frontend.java:822] Requesting prioritized load of table(s): default.comm_simeod_raw

i just do refresh whenever any partition is added in the table.even if i run the query on old data same problem comes.

[#7]

Does it happen all the time?

How many partitions and files in HDFS do you have for those tables that have the issue? How many tables affected? Are they all having large number of partitions and tables?

[#8]

not all the times but most of the times, table size is not even 100 gb. there are 1200 partitions in the table. my question is when the table is loaded in catalog which i can see by going to that tab then what does it mean by loading ? and why its doing ?