I am unable to connect to hive by the hive/beeline command every few days and need to go into Ambari and restart the hive processes to fix that. This is the error message that I see on Ambari:
Connection failed on host {server hostname}:10000 (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/alerts/alert_hive_thrift_port.py", line 204, in execute ldap_password=ldap_password) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/hive_check.py", line 84, in check_thrift_port_sasl ...
CRIT for 4 hours
Licensed under the Apache License, Version 2.0.
See third-party tools/resources that Ambari uses and their respective authors
Connection failed on host {server hostname}:10000 (Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/alerts/alert_hive_thrift_port.py", line 204, in execute
ldap_password=ldap_password)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/hive_check.py", line 84, in check_thrift_port_sasl
timeout_kill_strategy=TerminateStrategy.KILL_PROCESS_TREE,
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 308, in _call
raise ExecuteTimeoutException(err_msg)
ExecuteTimeoutException: Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'"'"' ; beeline -n hive -u '"'"'jdbc:hive2://{server hostname}:10000/;transportMode=binary'"'"' -e '"'"';'"'"' 2>&1 | awk '"'"'{print}'"'"' | grep -i -e '"'"'Connected to:'"'"' -e '"'"'Transaction isolation:'"'"''' was killed due timeout after 60 seconds
)
I tried following the steps listed here https://community.hortonworks.com/questions/208872/hive-server-2-process-connection-failed.html. I was able to get through all of them except the disabling the database scans one, couldn't figure out how that's done. It could have something to do with the Hive-Zookeeper connection. I am using HDP 3.1, hive version 3.1.0 and zookeeper version 3.4.6. I can provide more info as well that might help resolve this issue.
