PROBLEM STATEMENT: There are frequent email alerts from HiveServer2 Metastore process which are misleading. When check the HiveServer2 Metastore process is up and running fine without issue.
ERROR: Below is sample email alert -
Ambari Alerts log output shows -
/var/log/ambari-server/ambari-alerts.log
ROOT CAUSE: The issue was with kerberos credentials for ambari-qa user.
RESOLUTION:
HiveServer2 Metastore alert is triggered from the node on which the service is installed. "ambari-qa" user is used by ambari to trigger the alert. If you are using kerberized cluster make sure you have "ambari-qa" ticket in place else false alert will be triggered.
You can use below command to check -
$su - ambari-qa $klist <--[make sure you have valid ticket] $hive --hiveconf hive.metastore.uris=thrift://<hive_mestastore_host>:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e 'show databases;'
Also check, sometimes you might hit - https://issues.apache.org/jira/browse/AMBARI-14424
