question Re: When I have to Refresh / Invalidate Metadata a table ? in Support Questions

When I have to Refresh / Invalidate Metadata a table ?

eMazarakis — Wed, 14 Aug 2019 19:03:19 GMT

Let's assume that I have a table test_tbl which was created through impala-shell.

I have a few questions :

REFRESH the table only when I add new data through HIVE or HDFS commands ? That is when I am doing insert into ...through impala-shell no need for refreshing ?
INVALIDATE METADATA of the table only when I change the structure of the table (add columns, drop partitions) through HIVE?
DROPping partitions of a table through impala-shell (i.e alter table .. drop partition .. purge). Do I have to do REFRESH or INVALIDATE METADATA?
DROPping partitions of a table through impala-shell . How can I compute the new stats of the partitioned table? Compute incremental stats OR Drop Incremental stats before dropping partition ?

Thanks in advance.

Re: When I have to Refresh / Invalidate Metadata a table ?

araujo — Wed, 14 Aug 2019 22:31:24 GMT

REFRESH the table only when I add new data through HIVE or HDFS commands ? That is when I am doing insert into ...through impala-shell no need for refreshing ?

Correct.

INVALIDATE METADATA of the table only when I change the structure of the table (add columns, drop partitions) through HIVE?

Correct. Or creating new tables through Hive.

DROPping partitions of a table through impala-shell (i.e alter table .. drop partition .. purge). Do I have to do REFRESH or INVALIDATE METADATA?

No.

DROPping partitions of a table through impala-shell . How can I compute the new stats of the partitioned table? Compute incremental stats OR Drop Incremental stats before dropping partition ?

The next time you run an incremental stats for a new partition Impala will update things correctly (e.g. the global row count)

Re: When I have to Refresh / Invalidate Metadata a table ?

EricL — Thu, 15 Aug 2019 00:27:03 GMT

For number 2, ANY changes outside of Impala, you will need INVALIDATE METADATA, or if new data added, then REFRESH will do.

Work is underway to improve it: https://issues.apache.org/jira/browse/IMPALA-3124

Cheers
Eric