Reply
Explorer
Posts: 45
Registered: ‎09-29-2016

Hive metastore not able to execute query on metastore table when it runs thousand of query at one go

[ Edited ]

Hi,

 

I am facing a weired problem across all the env. today i tried to add one column to hive table with cascade command to reflect in all existing partitions. it took 6 hours to update 4500 partitions.

 

even if i do compute incremenal stats from impala for the first time it took 6 hours to do that. 

the hive metastore is in oracle. 

 

looks like the query is not getting executed at the speed and somewhere its going in wait at database oracle server.

or it may also possible that hive metastore is not sending all the queries to oracle .

 

what could be the reason, i am not sure whats going on, any help will be appreciated.

Cloudera Employee
Posts: 279
Registered: ‎03-23-2015

Re: Hive metastore not able to execute query on metastore table when it runs thousand of query at on

It is hard to tell what's going on. Suggest to check for HS2 or HMS server log, and collect jstack for both HS2 and HMS to determine what hive was doing at the time. (suspect that bottleneck is at HMS or Oracle Level).
Explorer
Posts: 45
Registered: ‎09-29-2016

Re: Hive metastore not able to execute query on metastore table when it runs thousand of query at on

Hello,

 

 I have done all the analysys and found below things :

 

1. involved DBA to check if there is any problem at Oracle where my metastore is, but can see there is no problem 

all of the queries have finished normally,did not see any thread contention or wait time also.

2. then i tried to put logger level to all in Hive log4j properties and i was able to see all the actual queries being run to  hive meta store. and here i can see there is no problem becuase all the queries are being executed in ms of times.

so it looks that there is no problem at HMS Oracle.

 

also i tried doing compute incremental stats from Impala and same problem i can see.

 

 

Announcements