- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
When I have to Refresh / Invalidate Metadata a table ?
- Labels:
-
Apache Hive
-
Apache Impala
Created on ‎08-14-2019 12:00 PM - edited ‎08-14-2019 12:03 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Let's assume that I have a table test_tbl which was created through impala-shell.
I have a few questions :
- REFRESH the table only when I add new data through HIVE or HDFS commands ? That is when I am doing insert into ...through impala-shell no need for refreshing ?
- INVALIDATE METADATA of the table only when I change the structure of the table (add columns, drop partitions) through HIVE?
- DROPping partitions of a table through impala-shell (i.e alter table .. drop partition .. purge). Do I have to do REFRESH or INVALIDATE METADATA?
- DROPping partitions of a table through impala-shell . How can I compute the new stats of the partitioned table? Compute incremental stats OR Drop Incremental stats before dropping partition ?
Thanks in advance.
Created ‎08-14-2019 03:31 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
REFRESH the table only when I add new data through HIVE or HDFS commands ? That is when I am doing insert into ...through impala-shell no need for refreshing ?
Correct.
- INVALIDATE METADATA of the table only when I change the structure of the table (add columns, drop partitions) through HIVE?
Correct. Or creating new tables through Hive.
- DROPping partitions of a table through impala-shell (i.e alter table .. drop partition .. purge). Do I have to do REFRESH or INVALIDATE METADATA?
No.
- DROPping partitions of a table through impala-shell . How can I compute the new stats of the partitioned table? Compute incremental stats OR Drop Incremental stats before dropping partition ?
The next time you run an incremental stats for a new partition Impala will update things correctly (e.g. the global row count)
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Created ‎08-14-2019 03:31 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
REFRESH the table only when I add new data through HIVE or HDFS commands ? That is when I am doing insert into ...through impala-shell no need for refreshing ?
Correct.
- INVALIDATE METADATA of the table only when I change the structure of the table (add columns, drop partitions) through HIVE?
Correct. Or creating new tables through Hive.
- DROPping partitions of a table through impala-shell (i.e alter table .. drop partition .. purge). Do I have to do REFRESH or INVALIDATE METADATA?
No.
- DROPping partitions of a table through impala-shell . How can I compute the new stats of the partitioned table? Compute incremental stats OR Drop Incremental stats before dropping partition ?
The next time you run an incremental stats for a new partition Impala will update things correctly (e.g. the global row count)
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Created ‎08-14-2019 05:27 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Work is underway to improve it: https://issues.apache.org/jira/browse/IMPALA-3124
Cheers
Eric
