Support Questions

Find answers, ask questions, and share your expertise

HiveServer 2 goes down

avatar

Hive server 2 goes down in our environment within 5 minutes of bringing it up with the following error, any thoughts

2016-12-22 13:17:31,526 FATAL [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(236)) - Unable to create a znode for this server instance
java.lang.Exception: Max znode creation wait time: 120s exhausted
        at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:225)
        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417)
        at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78)
        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654)
        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
2016-12-22 13:17:31,531 INFO  [main]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2
2016-12-22 13:17:31,531 INFO  [main]: thrift.ThriftCLIService (ThriftCLIService.java:stop(199)) - Thrift server has stopped
2016-12-22 13:17:31,531 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2016-12-22 13:17:31,532 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2016-12-22 13:17:31,532 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2016-12-22 13:17:41,533 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped.
2016-12-22 13:17:41,534 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped.
2016-12-22 13:17:41,544 INFO  [main]: zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x358a9c9199506a2 closed
2016-12-22 13:17:41,545 INFO  [main]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper.
2016-12-22 13:17:41,545 WARN  [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(442)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds
java.lang.Exception: Max znode creation wait time: 120s exhausted
        at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:225)
        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417)
        at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78)
        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654)
        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
2016-12-22 13:17:41,546 INFO  [main-EventThread]: zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
2016-12-22 13:18:33,625 INFO  [Thread-4]: server.HiveServer2 (HiveStringUtils.java:run(709)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at abc.solutions.local/172.16.3.196
************************************************************/
2016-12-22 13:18:33,631 INFO  [Thread-7]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2
2016-12-22 13:18:33,632 INFO  [Thread-7]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper
1 ACCEPTED SOLUTION

avatar
Super Guru
@ARUN

it looks hiveserver2 is not able to create zk node in zookeeper, it could be issue at zookeeper side

could you please check whether zk node got created or not using

/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server localhost:2181

ls /hiveserver2

see if you have issue with zookeeper server.

View solution in original post

11 REPLIES 11

avatar
Contributor

I am also facing same issue. After login into zk command like. I could only see below content. I can't see HS2

[zk: localhost:2181(CONNECTED) 10] ls /

[registry, ambari-metrics-cluster, zookeeper, zk_smoketest, rmstore]

[zk: localhost:2181(CONNECTED) 11]

Could you Please help me on this?

When I was starting HiveServer2 first time after installation I got below error. Tried multiple times but HS2 was never able to start.

Traceback (most recent call last):

File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 54, in wrapper

return function(*args, **kwargs)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 189, in wait_for_znode

raise Fail(format("ZooKeeper node /{hive_server2_zookeeper_namespace} is not ready yet"))

Fail: ZooKeeper node /hiveserver2 is not ready yet

The above exception was the cause of the following exception:

Traceback (most recent call last):

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 137, in <module>

HiveServer().execute()

File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute

method(env)

File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart

self.start(env, upgrade_type=upgrade_type)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 53, in start

hive_service('hiveserver2', action = 'start', upgrade_type=upgrade_type)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 101, in hive_service

wait_for_znode()

File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 62, in wrapper

return function(*args, **kwargs)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 189, in wait_for_znode

raise Fail(format("ZooKeeper node /{hive_server2_zookeeper_namespace} is not ready yet"))

resource_management.core.exceptions.Fail: ZooKeeper node /hiveserver2 is not ready yet

avatar
Explorer

i got the same problem.

When I reinstalled hive, everything worked.