Frequent Ambari alert on HiveServer2 Metastore Process

[#1]

PROBLEM STATEMENT: There are frequent email alerts from HiveServer2 Metastore process which are misleading. When check the HiveServer2 Metastore process is up and running fine without issue.

ERROR: Below is sample email alert -

Ambari Alerts log output shows -

/var/log/ambari-server/ambari-alerts.log

ROOT CAUSE: The issue was with kerberos credentials for ambari-qa user.

RESOLUTION:

HiveServer2 Metastore alert is triggered from the node on which the service is installed. "ambari-qa" user is used by ambari to trigger the alert. If you are using kerberized cluster make sure you have "ambari-qa" ticket in place else false alert will be triggered.

You can use below command to check -

$su - ambari-qa 
$klist   <--[make sure you have valid ticket]

$hive --hiveconf hive.metastore.uris=thrift://<hive_mestastore_host>:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e 'show databases;' 

Also check, sometimes you might hit - https://issues.apache.org/jira/browse/AMBARI-14424