When I have to Refresh / Invalidate Metadata a table ?

Question

Let's assume that I have a table   test_tbl which was created through impala-shell.I have a few questions :REFRESH the table only when I add new data through HIVE or HDFS commands ? That is when I am doing insert into ...through impala-shell no need for refreshing ?INVALIDATE METADATA  of the table only when I change the structure of the table (add columns, drop partitions) through HIVE? DROPping partitions of a table through impala-shell (i.e alter table  .. drop partition .. purge).  Do I have to do REFRESH or INVALIDATE METADATA?DROPping partitions of a table through impala-shell . How can I compute the new stats of the partitioned table?  Compute incremental stats OR Drop Incremental stats before dropping partition ? Thanks in advance.

araujo · Accepted Answer

REFRESH the table only when I add new data through HIVE or HDFS commands ? That is when I am doing insert into ...through impala-shell no need for refreshing ?Correct.INVALIDATE METADATA  of the table only when I change the structure of the table (add columns, drop partitions) through HIVE? Correct. Or creating new tables through Hive.DROPping partitions of a table through impala-shell (i.e alter table  .. drop partition .. purge).  Do I have to do REFRESH or INVALIDATE METADATA?No.DROPping partitions of a table through impala-shell . How can I compute the new stats of the partitioned table?  Compute incremental stats OR Drop Incremental stats before dropping partition ?The next time you run an incremental stats for a new partition Impala will update things correctly (e.g. the global row count)

EricL · Answer

For number 2, ANY changes outside of Impala, you will need INVALIDATE METADATA, or if new data added, then REFRESH will do.

Work is underway to improve it: https://issues.apache.org/jira/browse/IMPALA-3124

Cheers
Eric

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.