HiveServer 2 goes down

[#1]

Hive server 2 goes down in our environment within 5 minutes of bringing it up with the following error, any thoughts

2016-12-22 13:17:31,526 FATAL [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(236)) - Unable to create a znode for this server instance
java.lang.Exception: Max znode creation wait time: 120s exhausted
        at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:225)
        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417)
        at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78)
        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654)
        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
2016-12-22 13:17:31,531 INFO  [main]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2
2016-12-22 13:17:31,531 INFO  [main]: thrift.ThriftCLIService (ThriftCLIService.java:stop(199)) - Thrift server has stopped
2016-12-22 13:17:31,531 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2016-12-22 13:17:31,532 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2016-12-22 13:17:31,532 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2016-12-22 13:17:41,533 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped.
2016-12-22 13:17:41,534 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped.
2016-12-22 13:17:41,544 INFO  [main]: zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x358a9c9199506a2 closed
2016-12-22 13:17:41,545 INFO  [main]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper.
2016-12-22 13:17:41,545 WARN  [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(442)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds
java.lang.Exception: Max znode creation wait time: 120s exhausted
        at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:225)
        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417)
        at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78)
        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654)
        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
2016-12-22 13:17:41,546 INFO  [main-EventThread]: zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
2016-12-22 13:18:33,625 INFO  [Thread-4]: server.HiveServer2 (HiveStringUtils.java:run(709)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at abc.solutions.local/172.16.3.196
************************************************************/
2016-12-22 13:18:33,631 INFO  [Thread-7]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2
2016-12-22 13:18:33,632 INFO  [Thread-7]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper

[#2]

I am also facing same issue. After login into zk command like. I could only see below content. I can't see HS2

[zk: localhost:2181(CONNECTED) 10] ls /

[registry, ambari-metrics-cluster, zookeeper, zk_smoketest, rmstore]

[zk: localhost:2181(CONNECTED) 11]

Could you Please help me on this?

When I was starting HiveServer2 first time after installation I got below error. Tried multiple times but HS2 was never able to start.

Traceback (most recent call last):

File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 54, in wrapper

return function(*args, **kwargs)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 189, in wait_for_znode

raise Fail(format("ZooKeeper node /{hive_server2_zookeeper_namespace} is not ready yet"))

Fail: ZooKeeper node /hiveserver2 is not ready yet

The above exception was the cause of the following exception:

Traceback (most recent call last):

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 137, in <module>

HiveServer().execute()

File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute

method(env)

File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart

self.start(env, upgrade_type=upgrade_type)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 53, in start

hive_service('hiveserver2', action = 'start', upgrade_type=upgrade_type)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 101, in hive_service

wait_for_znode()

File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 62, in wrapper

return function(*args, **kwargs)

File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 189, in wait_for_znode

raise Fail(format("ZooKeeper node /{hive_server2_zookeeper_namespace} is not ready yet"))

resource_management.core.exceptions.Fail: ZooKeeper node /hiveserver2 is not ready yet

[#3]

i got the same problem.

When I reinstalled hive, everything worked.