HiveServer2 Not Starting : ZooKeeper node does not exist: /hiveserver2

[#1]

I am getting below error while starting HiveServer2 after installation.

Node does not exist: /hiveserver2

Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 54, in wrapper
    return function(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 189, in wait_for_znode
    raise Fail(format("ZooKeeper node /{hive_server2_zookeeper_namespace} is not ready yet")) 

Fail: ZooKeeper node /hiveserver2 is not ready yet

Here are some additional details around error :

2018-09-13 16:25:26,594 - call['ambari-sudo.sh su hive -l -s /bin/bash -c 'cat /var/run/hive/hive-server.pid 1>/tmp/tmpIldfIy 2>/tmp/tmpDcXaXg''] {'quiet': False}
2018-09-13 16:25:26,638 - call returned (1, '')
2018-09-13 16:25:26,638 - Execution of 'cat /var/run/hive/hive-server.pid 1>/tmp/tmpIldfIy 2>/tmp/tmpDcXaXg' returned 1. cat: /var/run/hive/hive-server.pid: No such file or directory

2018-09-13 16:25:26,638 - get_user_call_output returned (1, u'', u'cat: /var/run/hive/hive-server.pid: No such file or directory')
2018-09-13 16:25:26,639 - call['ambari-sudo.sh su hive -l -s /bin/bash -c 'hive --config /usr/hdp/current/hive-server2/conf/ --service metatool -listFSRoot' 2>/dev/null | grep hdfs:// | cut -f1,2,3 -d '/' | grep -v 'hdfs://seidevdsmastervm01.tsudev.seic.com:8020' | head -1'] {}
2018-09-13 16:25:33,149 - call returned (0, '')
2018-09-13 16:25:33,149 - Execute['/var/lib/ambari-agent/tmp/start_hiveserver2_script /var/log/hive/hive-server2.out /var/log/hive/hive-server2.err /var/run/hive/hive-server.pid /usr/hdp/current/hive-server2/conf/ /etc/tez/conf'] {'environment': {'HIVE_BIN': 'hive', 'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_112', 'HADOOP_HOME': u'/usr/hdp/current/hadoop-client'}, 'not_if': 'ls /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps -p  >/dev/null 2>&1', 'user': 'hive', 'path': [u'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/usr/hdp/current/hive-server2/bin:/usr/hdp/3.0.0.0-1634/hadoop/bin']}
2018-09-13 16:25:33,196 - Execute['/usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-server2/lib/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://seidevdsmastervm01.tsudev.seic.com/hive?createDatabaseIfNotExist=true' hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 'try_sleep': 10}
2018-09-13 16:25:33,476 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server worker1_node:2181,worker2_node.com:2181,master_node.com:2181 ls /hiveserver2 | grep 'serverUri=''] {} 

2018-09-13 16:25:34,068 - call returned (1, 'Node does not exist: /hiveserver2')

Error Message : 2018-09-13 16:25:34,068 - call returned (1, 'Node does not exist: /hiveserver2')

I have checked zk command line on master node where HS2 in configured and could not find HS2 there. Here is output :

[zk: localhost:2181(CONNECTED) 1] ls /

[registry, ambari-metrics-cluster, zookeeper, zk_smoketest, rmstore]

[zk: localhost:2181(CONNECTED) 2]

I am using HDP3.0, This is a 3 node cluster (1 Master + 2 Worker). Hive metastore (MySQL 5.7) is installed on master node and HiveServer2 is also configured on master node. All Machines have Oracle Linux 7.

[#2]

Found the reason for my issue.....

After long time looking into logs and activating the DEBUG log level for hive, finally found that:

DEBUG [main]: component.AbstractLifeCycle (:()) - starting ServerConnector@748904e8{HTTP/1.1,[http/1.1]}{0.0.0.0:10002}
2018-10-25T10:50:05,352 ERROR [main]: server.HiveServer2 (HiveServer2.java:start(740)) - Error starting Web UI: 
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_112]
    at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_112]
    at sun.nio.ch.Net.bind(Net.java:425) ~[?:1.8.0_112]

Issue was that the Hive WebUI standard port 10002 was already used by a different program. I changed the port in Hive config. Now Hive is working fine.

Maybe, same thing for you @Shantanu Sharma

[#3]

Hello @Björn List,

I'm facing the same problem for an emergency situation.

Can you show me please how to find the right port to set instead of "10002" ?

Which port should we put ?

Thank you in advance 🙂

[#4]

Hi @Ayoub Bounfour,

you just need to find a port which is currently not used 🙂

What I did:

1) Select a port number randomly - In my case I added +100 to the port number, so 10102 was my desired port.

2) check if this port is already used. In my environment, I executed following command:

#	netstat -tulpn | grep 10102

If the result is empty, it means there is currently no service listening at this port. So, it seems to be free. (Attention! It still could be that another service is using that port, but it is just not running at the time you run this command.)

3) If port you selected in step 1 is not used (empty result of command from step 2) change the hive config in ambari WebUI. If result of step 2 was not empty, start with step 1 again and select randomly a different port.

Hope this helps.

regards,

Björn

[#5]

Hi @Björn List,

Thank you so much 🙂

By the way, I found that Accumulo was in the same port of Hive, so I disabled The first one to run hiveserver2.

Best regards,

Ayoub

[#6]

@er_sharma_shant @jsensharma Can you guys please tell me how to resolve hiveserver2 start issue by adding the znode name for hiveserver2 in zkcli shell?.. I have hiveserver2 instance created in zkcli shell, but it does not have znode name because of which hiveserver2 is failing to start.

Welcome to ZooKeeper!
2022-02-03 02:31:36,504 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1013] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2022-02-03 02:31:36,592 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@856] - Socket connection established, initiating session, client: /127.0.0.1:36762, server: localhost/127.0.0.1:2181
2022-02-03 02:31:36,609 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1273] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x27ebd79aecc014c, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[cluster, registry, controller, brokers, storm, infra-solr, zookeeper, hbase-unsecure, hadoop-ha, tracers, admin, isr_change_notification, log_dir_event_notification, accumulo, controller_epoch, hiveserver2, hiveserver2-leader, druid, rmstore, atsv2-hbase-unsecure, consumers, ambari-metrics-cluster, latest_producer_id_block, config]

[zk: localhost:2181(CONNECTED) 1] ls /hiveserver2
[]
[zk: localhost:2181(CONNECTED) 2] 

Any help would be much appreciated!!!. Thank you