Support Questions
Find answers, ask questions, and share your expertise

Hive Metastore failure urgent help !!!!

Contributor

Hi

Hive metastore is giving error.Tried restarting the Metsatore,Hive and the cluster also .Still the error is there .

Metastore on hklpaprad12.hk.xxxxxxxx.com failed (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py", line 190, in execute timeout=int(check_command_timeout) ) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 285, in _call raise ExecuteTimeoutException(err_msg) ExecuteTimeoutException: Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/tss/bin:/opt/tss/itrs:/opt/tss/post.d:/opt/VRTSvcs/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/root/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/sbin/:/usr/hdp/current/hive-metastore/bin'"'"' ; export HIVE_CONF_DIR='"'"'/usr/hdp/current/hive-metastore/conf/conf.server'"'"' ; hive --hiveconf hive.metastore.uris=thrift://hklpaprad12.hk.xxxxxxxx.com:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e '"'"'show databases;'"'"''' was killed due timeout after 60 seconds )

Any help appreciated

Thanks inadvance

5 REPLIES 5

could you please post hive metastore logs

Guru

check if the metastore is on :

hklpaprad12.hk.xxxxxxxx.com# netstat -anpe | grep 9083 | grep LISTEN

you should have one line, if not then metastore is off.

Like @Rajkumar Singh said, then look or post at metastore logs (if you post then use the "code" tag please)

Contributor

Hi All,

Looks like only my Hive Cli is not working .I am able to access Hive through beeline works file .Because the CLI is hanging Ambari is showing metastore critical.We dont have an option to use beeline cz all our code is written to use hive cli .

Any inputs.

@Rajkumar Singh

@Laurent Edel

@Shihab as per your comment you are getting metastore critical alert, could you please check metastore logs.

@Shihab

Do you have Tez jobs running in the setup for adhoc queries? Are the queues configured as documented here:

Using Tez with Hive Especially, the following:

<property>
     <name>hive.server2.tez.default.queues</name>
     <value>default</value>
</property>
 
<property>
     <name>hive.server2.tez.sessions.per.default.queue</name>
     <value>1</value>
</property>

If there are going to simultaneous connections, more queues need to be defined and each queue should have 2-5 sessions.