Reply
Explorer
Posts: 30
Registered: ‎01-31-2016

Impala/Hive metadata problem

Hi,

I have a partitioned table in  hive and I know that if I want to use this table in impala, i have to call "invalidate metadata tablename" command to retrieve tables metadata to catalog servers.

 
Now we want to clean catalog server's metadata without touching hive content but we couldnt find any resource on web.
Cloudera Employee
Posts: 50
Registered: ‎12-11-2015

Re: Impala/Hive metadata problem

[ Edited ]

Catalog Server doesnt persist metadata in disk. It retrieves the metadata from HiveMetaStore and stores it in memory. So everytime you restart catalog service the metadata that catalog holds gets cleaned

 

 Suppose if you want to get the changes made on a specific table in hive, you can do that using below command

 

INVALIDATE METADATA [[db_name.]table_name] 

Invalidate metadata db.table_name will reset the metadata related to the table in impala

 

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_invalidate_metadata.html#inval...

Contributor
Posts: 66
Registered: ‎12-30-2015

Re: Impala/Hive metadata problem

@venkatsambath already answered to your question.

 

One thing I'd like to add is that please run simple `SELECT` query for the table you want to reload metadata after running `INAVLIDATE META`. By doing that, you can trigger loading metadata in Impala.

 

Gatsby

Explorer
Posts: 30
Registered: ‎01-31-2016

Re: Impala/Hive metadata problem

Hi,

 

When we restart  impala we do not want to see anything from "show tables" in impala. But we see all the tables. How can we clear all meta info in impala without deleting them in hive?

Cloudera Employee
Posts: 82
Registered: ‎12-07-2015

Re: Impala/Hive metadata problem

Hi scobanx,

I'm not sure I understood your question correctly. Can you elaborate on
what you're trying to do?

Impala displays metadata as it is stored in the Hive metastore, and there
is no functionality that I'm aware of, that would let you break that
synchronization on purpose. Without metadata, Impala would not be able to
run queries against these tables.

Cheers, Lars
Explorer
Posts: 30
Registered: ‎01-31-2016

Re: Impala/Hive metadata problem

Assume we added impala service and started impala, in this state when we execute "show tables" from impala shell we see no tables. As soon as we execute "invalidate metadata" and then "show tables" we see all tables in hive metastore. I know impala does not store any metadata info on disk, so I assume when we restart impala we should return to empty "show tables" output, but this is not the case. 

 

What I understand from your post is, this is not possible right?

Cloudera Employee
Posts: 82
Registered: ‎12-07-2015

Re: Impala/Hive metadata problem

If you start Impala and then run show tables, it should fetch them from the
metastore and display them. Can you post detailed steps how to reproduce
this behavior? How do you deploy Impala? How do you restart it? What
version of what components are involved?

Thank you, Lars
Master
Posts: 430
Registered: ‎07-01-2015

Re: Impala/Hive metadata problem

But whats the point to not showing tables? 

Are you truing to hide tables before users? Then you can do thaht by grants in Sentry.