Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

"invalidate metadata" do not work

Solved Go to solution

Re: "invalidate metadata" do not work

Contributor

You are right. hivecli wont reach the metastore if it doesnt have proper configuration files.

 

Did the node inwhich you ran the hiveCLI had proper hive gateway assigned to it? If not, the table that you created would have been created on the derby database(embedded DB of HMS) and this would inturn makes the changes invisible to impala, because impala catalog always grabs the metadata from the remote metastore.

 

This also clarifies that other behaviour that you noticed. i.e creating database on one node was not being reflected on the hivecli opened in another node because the metadata are being added on the embedded DB

Re: "invalidate metadata" do not work

New Contributor

There's a Hive Gateway on each node. 

Re: "invalidate metadata" do not work

New Contributor

1. create a table in Hive and run invalidate metadata in Impala. Does this work?

This is what I've done several times (create table/database in Hive; then run invalidate metadata in Impala). And no. It doesn't work.

 

2. Check the impalad.INFO and catalogd.INFO for any exceptions/errors

 

catalogd.ERROR  includes exceptions concerning the missing tables. But there's no additional information about the origin of the problem : 

E1024 15:55:53.764930 10217 catalog-server.cc:76] TableNotFoundException: Table not found: new_db_from_hive.new_table_from_hive

E1024 15:59:52.447249 10217 catalog-server.cc:76] TableNotFoundException: Table not found: new_db_from_hive2nd.new_table_from_hive

E1024 16:18:37.696986 14116 catalog-server.cc:76] TableNotFoundException: Table not found: new_db_from_hive.new_table_from_hive

E1024 16:30:49.233502 10217 catalog-server.cc:76] TableNotFoundException: Table not found: new_db_from_hive3rd.new_table_from_hive

 

In impalad.INFO, I found some lines that *may* be of interest (?) :

I1024 18:50:41.929322 10036 Frontend.java:866] analyze query invalidate metadata

I1024 18:50:41.950764 10036 impala-server.cc:1430] Waiting for catalog version: 34 current version: 26

I1024 18:50:45.444826 10036 impala-beeswax-server.cc:307] get_results_metadata(): query_id=5243589279189211:df729a8d9a1687a4

I1024 18:50:45.445307 10036 impala-beeswax-server.cc:349] close(): query_id=5243589279189211:df729a8d9a1687a4

I1024 18:50:45.445310 10036 impala-server.cc:915] UnregisterQuery(): query_id=5243589279189211:df729a8d9a1687a4

I1024 18:50:45.445314 10036 impala-server.cc:1001] Cancel(): query_id=5243589279189211:df729a8d9a1687a4

I1024 18:50:45.446462 10036 impala-beeswax-server.cc:172] query(): query=invalidate metadata

 

I don't know if this variation of version in the catalog is normal. 

 

3. Keep in mind that invalidate metadata is asynchronous, i.e. it returns immediately and then in the background the catalog objects are sent to the impalad nodes. This could take some time, so the question is if the db shows up eventually (after a few secs) or never.

There are few data in Hive and Impala. 
And even if I try looking at the existing tables in Impala much later after I ran the "invalidate metadata" query, they still do not appear.