Support Questions
Find answers, ask questions, and share your expertise

Hive Server2 service getting stopped

Hive Server2 service getting stopped

Contributor

i have seen a regular behavior of hive server2 getting stopped while we are using hive from Qliksense or multiple query from Zeppelin. I am using HDP-2.4.2.0 .I am getting the following error.

Connection failed on host XXX:10000 (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py", line 200, in execute check_command_timeout=int(check_command_timeout)) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/hive_check.py", line 74, in check_thrift_port_sasl timeout=check_command_timeout) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) Fail: Execution of '! beeline -u 'jdbc:hive2://XXXXX:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'' returned 1. stdin: is not a tty stdin: is not a tty Error: Could not open client transport with JDBC Uri: jdbc:hive2://xxxx:10000/;transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0) Error: Could not open client transport with JDBC Uri: jdbc:hive2://lxdb4231.dc.corp.telstra.com:10000/;transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0)

7 REPLIES 7
Highlighted

Re: Hive Server2 service getting stopped

Hi @bandhu gupta

Can you please share the hiveserver2 logs?

Highlighted

Re: Hive Server2 service getting stopped

Contributor

This is the hive error shown below.

21545:2016-09-21 18:34:04,785 ERROR [HiveServer2-Handler-Pool: Thread-78]: metadata.Hive (Hive.java:getTable(1121)) - Table XXXXX not found: default.XXXXX table not found 21622:2016-09-21 18:34:05,004 ERROR [HiveServer2-Handler-Pool: Thread-78]: hdfs.KeyProviderCache (KeyProviderCache.java:createKeyProviderURI(87)) - Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!

Highlighted

Re: Hive Server2 service getting stopped

Expert Contributor

A highly possible cause can be HiveServer2 running out of memory.

The defaults from recent Ambari versions are quite low: 512MB of Java heap.

Check /var/log/hive/hiveserver2.log and /var/log/hive/hive-server2.out for any indication of such an issue.

And increase HiveServer2 Heap using Ambari from the default 512MB.

2048MB would be a start:

7829-screen-shot-2016-09-21-at-141832.png

Highlighted

Re: Hive Server2 service getting stopped

Contributor

I have increased the heap size to 1638 MB , Metastore heap size to 8052 Mb , Client heap size to 1024 mb but still i am getting this random behavior . Please suggest.

Highlighted

Re: Hive Server2 service getting stopped

Super Guru
@bandhu gupta

Please share hiveserver2 logs. It is likely a memory issue. Look for "Out of memory" error in your logs. Should be in the stack trace towards the end. Also, please share your hiveserver 2 config. There are also memory settings in hive-env.sh but you can set these from Ambari.

Highlighted

Re: Hive Server2 service getting stopped

Contributor

<property> <name>hive.security.metastore.authorization.manager</name> <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value> </property> <property> <name>hive.server2.allow.user.substitution</name> <value>true</value> </property> <property> <name>hive.server2.authentication</name> <value>NONE</value> </property> <property> <name>hive.server2.authentication.spnego.keytab</name> <value>HTTP/_HOST@EXAMPLE.COM</value> </property> <property> <name>hive.server2.authentication.spnego.principal</name> <value>/etc/security/keytabs/spnego.service.keytab</value> </property> <property> <name>hive.server2.enable.doAs</name> <value>false</value> </property> <property> <name>hive.server2.logging.operation.enabled</name> <value>true</value> </property> <property> <name>hive.server2.logging.operation.log.location</name> <value>/tmp/hive/operation_logs</value> </property> <property> <name>hive.server2.support.dynamic.service.discovery</name> <value>true</value> </property> <property> <name>hive.server2.table.type.mapping</name> <value>CLASSIC</value> </property> <property> <name>hive.server2.tez.default.queues</name> <value>default</value> </property> <property> <name>hive.server2.tez.initialize.default.sessions</name> <value>false</value> </property> <property> <name>hive.server2.tez.sessions.per.default.queue</name> <value>1</value> </property> <property> <name>hive.server2.thrift.http.path</name> <value>cliservice</value> </property> <property> <name>hive.server2.thrift.http.port</name> <value>10001</value> </property> <property> <name>hive.server2.thrift.max.worker.threads</name> <value>500</value> </property> <property> <name>hive.server2.thrift.port</name> <value>10000</value> </property> <property> <name>hive.server2.thrift.sasl.qop</name> <value>auth</value> </property> <property> <name>hive.server2.transport.mode</name> <value>binary</value> </property> <property> <name>hive.server2.use.SSL</name> <value>false</value> </property> <property> <name>hive.server2.zookeeper.namespace</name> <value>hiveserver2</value> </property>

Highlighted

Re: Hive Server2 service getting stopped

Contributor

did you find out the cause of this issue. I'm seeing the same problem

Don't have an account?