question Re: HiveServer 2 goes down in Archives of Support Questions (Read Only)

HiveServer 2 goes down

arunpoy — Fri, 23 Dec 2016 15:02:46 GMT

Hive server 2 goes down in our environment within 5 minutes of bringing it up with the following error, any thoughts

2016-12-22 13:17:31,526 FATAL [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(236)) - Unable to create a znode for this server instance
java.lang.Exception: Max znode creation wait time: 120s exhausted
        at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:225)
        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417)
        at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78)
        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654)
        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
2016-12-22 13:17:31,531 INFO  [main]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2
2016-12-22 13:17:31,531 INFO  [main]: thrift.ThriftCLIService (ThriftCLIService.java:stop(199)) - Thrift server has stopped
2016-12-22 13:17:31,531 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2016-12-22 13:17:31,532 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2016-12-22 13:17:31,532 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2016-12-22 13:17:41,533 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped.
2016-12-22 13:17:41,534 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped.
2016-12-22 13:17:41,544 INFO  [main]: zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x358a9c9199506a2 closed
2016-12-22 13:17:41,545 INFO  [main]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper.
2016-12-22 13:17:41,545 WARN  [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(442)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds
java.lang.Exception: Max znode creation wait time: 120s exhausted
        at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:225)
        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417)
        at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78)
        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654)
        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
2016-12-22 13:17:41,546 INFO  [main-EventThread]: zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
2016-12-22 13:18:33,625 INFO  [Thread-4]: server.HiveServer2 (HiveStringUtils.java:run(709)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at abc.solutions.local/172.16.3.196
************************************************************/
2016-12-22 13:18:33,631 INFO  [Thread-7]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2
2016-12-22 13:18:33,632 INFO  [Thread-7]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper

Re: HiveServer 2 goes down

rajkumar_singh — Fri, 23 Dec 2016 15:08:44 GMT

@ARUN

it looks hiveserver2 is not able to create zk node in zookeeper, it could be issue at zookeeper side

could you please check whether zk node got created or not using

/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server localhost:2181

ls /hiveserver2

see if you have issue with zookeeper server.

Re: HiveServer 2 goes down

sgowda — Fri, 23 Dec 2016 15:11:34 GMT

@ARUN

Looks like there is an issue with zookeeper. Can you please share zookeeper logs and look for any errors

Re: HiveServer 2 goes down

arunpoy — Fri, 23 Dec 2016 15:18:27 GMT

hi @Rajkumar Singh, this is what i see in the ZK

[zk: localhost(CONNECTED) 1] ls /hiveserver2 [serverUri=abcd.solutions.local:10000;version=1.2.1000.2.4.0.0-169;sequence=0000000000]

Re: HiveServer 2 goes down

rajkumar_singh — Fri, 23 Dec 2016 15:22:46 GMT

@ARUN

is this a standalone zk or running zk quorum?

Re: HiveServer 2 goes down

arunpoy — Fri, 23 Dec 2016 15:26:28 GMT

@Rajkumar Singh

it is a quorum of 3 servers

Re: HiveServer 2 goes down

rajkumar_singh — Fri, 23 Dec 2016 15:34:26 GMT

could you please attach complete hiveserver2 logs, it looks hs2 is not able to create zknode, this could be kerberos issue if you running in secure env or there might be zookeeper database inconsisency.

to rule out zk database inconsistencies you can try

stop all services which are using zk

change ZooKeeper directory location after browsing ambari=> zookeeper=> conf, by default it is /hadoop/zookeeper

start all reservices

Re: HiveServer 2 goes down

arunpoy — Fri, 23 Dec 2016 15:45:07 GMT

@Rajkumar Singh, i removed the znode /hiveserver2 from zk after stopping hive service. restarted again.it is running fine for more than 20 mins. Let me see what happens. Thanks for your help

Re: HiveServer 2 goes down

vishwanath_voru — Fri, 17 Mar 2017 18:07:47 GMT

Login to zk command line using zkCli.sh and after logging in check the contents of Hiveserver2 znode and in my case it is [ ], empty.

Delete the HS2 znode using del command and restart HS2 again.

Re: HiveServer 2 goes down

liulixia — Tue, 09 May 2017 18:20:21 GMT

Yes, it is working after Delete the HS2 znode using del command and restart HS2.

Re: HiveServer 2 goes down

er_sharma_shant — Fri, 14 Sep 2018 08:51:59 GMT

I am also facing same issue. After login into zk command like. I could only see below content. I can't see HS2

[zk: localhost:2181(CONNECTED) 10] ls /

[registry, ambari-metrics-cluster, zookeeper, zk_smoketest, rmstore]

[zk: localhost:2181(CONNECTED) 11]

Could you Please help me on this?

When I was starting HiveServer2 first time after installation I got below error. Tried multiple times but HS2 was never able to start.

Traceback (most recent call last):

File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 54, in wrapper

return function(*args, **kwargs)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 189, in wait_for_znode

raise Fail(format("ZooKeeper node /{hive_server2_zookeeper_namespace} is not ready yet"))

Fail: ZooKeeper node /hiveserver2 is not ready yet

The above exception was the cause of the following exception:

Traceback (most recent call last):

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 137, in <module>

HiveServer().execute()

File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute

method(env)

File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart

self.start(env, upgrade_type=upgrade_type)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 53, in start

hive_service('hiveserver2', action = 'start', upgrade_type=upgrade_type)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 101, in hive_service

wait_for_znode()

File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 62, in wrapper

return function(*args, **kwargs)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 189, in wait_for_znode

raise Fail(format("ZooKeeper node /{hive_server2_zookeeper_namespace} is not ready yet"))

resource_management.core.exceptions.Fail: ZooKeeper node /hiveserver2 is not ready yet

Re: HiveServer 2 goes down

xiamu — Mon, 19 Jun 2023 12:00:01 GMT

i got the same problem.

When I reinstalled hive, everything worked.