hiveserver2 not starting in HDP 3.1.4

[#1]

Hello,

I am getting following error when trying to start Hive server =

2021-05-20 12:21:50,088 - Will retry 7 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2021-05-20 12:22:00,098 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server hdp-master.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-slave1.mydomain.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2021-05-20 12:22:00,780 - call returned (1, '')
2021-05-20 12:22:00,781 - Will retry 6 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2021-05-20 12:22:10,791 - Process with pid 12962 is not running. Stale pid file at /var/run/hive/hive-server.pid

I referred to similar threads at this forum but didn't help.

Pls suggest how to fix this issue thank you.

Regards,

Amey.

[#2]

@dmharshit 

A znode is supposed to be formed under zookeeper. Check for zookeeper logs during the HS2 start as to why it is or HS2 is not able to create znode.

[#3]

@tusharkathpal  Thanks.
zookeeper logs aren't showing any useful info I think =

2021-05-20 17:51:09,093 - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:os.name=Linux
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-1127.19.1.el7.x86_64
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:user.name=root
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:user.home=/root
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:user.dir=/var/lib/ambari-agent
2021-05-20 17:51:09,095 - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=hdp-master.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-slave1.mydomain.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@25f38edc
2021-05-20 17:51:09,120 - INFO  [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@1013] - Opening socket connection to server hdp-master.mydomain.com/10.200.104.187:2181. Will not attempt to authenticate using SASL (unknown error)
2021-05-20 17:51:09,190 - INFO  [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@856] - Socket connection established, initiating session, client: /10.200.104.188:37684, server: hdp-master.mydomain.com/10.200.104.187:2181
2021-05-20 17:51:09,211 - INFO  [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@1273] - Session establishment complete on server hdp-master.mydomain.com/10.200.104.187:2181, sessionid = 0x17988299eff0059, negotiated timeout = 30000
2021-05-20 17:51:19,799 - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.6-315--1, built on 08/23/2019 05:02 GMT
2021-05-20 17:51:19,801 - INFO  [main:Environment@100] - Client environment:host.name=hdp-slave1.mydomain.com
2021-05-20 17:51:19,801 - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_262
2021-05-20 17:51:19,804 - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2021-05-20 17:51:19,804 - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre
2021-05-20 17:51:19,804 - INFO  [main:Environment@100] - Client environment:java.class.path=/usr/hdp/current/zookeeper-client/bin/../build/classes:/usr/hdp/current/zookeeper-client/bin/../build/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/netty-3.10.5.Final.jar:/usr/hdp/current/zookeeper-client/bin/../lib/log4j-1.2.16.jar:/usr/hdp/current/zookeeper-client/bin/../lib/jline-0.9.94.jar:/usr/hdp/current/zookeeper-client/bin/../zookeeper-3.4.6.3.1.4.0-315.jar:/usr/hdp/current/zookeeper-client/bin/../src/java/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../conf::/usr/share/zookeeper/*

[#4]

After moving HIVE to another slave node, HIVE is now working but,

If I try to invoke hive shell from slave1 i.e. the problematic host, it tries to connect to the same host i.e. slave1.

I compared 'hive-site.xml' of slave1 & slave2. 
No difference.

Some suspect from the hive-site.xml which might causing slave1 to connect to slave1 after invoking hive shell =

 <property>
      <name>atlas.rest.address</name>
      <value>http://hdp-slave1.mydomain.com:21000</value>
    </property>

<property>
      <name>hive.cluster.delegation.token.store.zookeeper.connectString</name>
      <value>hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181</value>
    </property>

    <property>
      <name>hive.zookeeper.quorum</name>
      <value>hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181</value>
    </property>

[#5]

@dmharshit 

Check your Ambari UI--->HDFS-->Config-- look for the parameter hadoop.proxyuser.hive.hosts make sure it's the HS2 host is listed here.

Restart the stale config and let me know!

[#6]

@Shelton  Thanks for the reply.

I confirmed that it's HS2 listed in hadoop.proxyuser.hive.hosts i.e. slave2 host in my case which is running Hive server service by now.

[#7]

@dmharshit 
Is the problem still persistent?  Have you tried using Ambari REST API  to move the component and delete the component for the old host?
Please revert

[#8]

Problem is still there.

21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1127.19.1.el7.x86_64
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:user.name=eagledev
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/user1
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/user1
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@5ace1ed4
21/06/09 16:50:22 INFO zookeeper.ClientCnxn: Opening socket connection to server hdp-slave1.mydomain.com/10.200.104.188:2181. Will not attempt to authenticate using SASL (unknown error)
21/06/09 16:50:22 INFO zookeeper.ClientCnxn: Socket connection established to hdp-slave1.mydomain.com/10.200.104.188:2181, initiating session
21/06/09 16:50:22 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
21/06/09 16:50:22 INFO zookeeper.ClientCnxn: Session establishment complete on server hdp-slave1.mydomain.com/10.200.104.188:2181, sessionid = 0x279ef5fd2c3006b, negotiated timeout = 60000
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Session: 0x279ef5fd2c3006b closed
21/06/09 16:50:22 INFO zookeeper.ClientCnxn: EventThread shut down
org.apache.curator.CuratorZookeeperClient.startAdvancedTracer(Ljava/lang/String;)Lorg/apache/curator/drivers/OperationTrace;
Beeline version 3.1.0.3.1.4.0-315 by Apache Hive
0: jdbc:hive2://hdp-slave1.mydomain.com:2 (closed)>