Support Questions
Find answers, ask questions, and share your expertise

hiveserver2 not starting in HDP 3.1.4

Explorer

Hello,

I am getting following error when trying to start Hive server =

 

2021-05-20 12:21:50,088 - Will retry 7 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2021-05-20 12:22:00,098 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server hdp-master.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-slave1.mydomain.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2021-05-20 12:22:00,780 - call returned (1, '')
2021-05-20 12:22:00,781 - Will retry 6 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2021-05-20 12:22:10,791 - Process with pid 12962 is not running. Stale pid file at /var/run/hive/hive-server.pid

 

 

I referred to similar threads at this forum but didn't help.

Pls suggest how to fix this issue thank you.

Regards,

Amey.

7 REPLIES 7

Rising Star

@dmharshit 

 

A znode is supposed to be formed under zookeeper. Check for zookeeper logs during the HS2 start as to why it is or HS2 is not able to create znode.

 

Explorer

@tusharkathpal  Thanks.
zookeeper logs aren't showing any useful info I think =

2021-05-20 17:51:09,093 - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:os.name=Linux
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-1127.19.1.el7.x86_64
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:user.name=root
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:user.home=/root
2021-05-20 17:51:09,094 - INFO  [main:Environment@100] - Client environment:user.dir=/var/lib/ambari-agent
2021-05-20 17:51:09,095 - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=hdp-master.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-slave1.mydomain.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@25f38edc
2021-05-20 17:51:09,120 - INFO  [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@1013] - Opening socket connection to server hdp-master.mydomain.com/10.200.104.187:2181. Will not attempt to authenticate using SASL (unknown error)
2021-05-20 17:51:09,190 - INFO  [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@856] - Socket connection established, initiating session, client: /10.200.104.188:37684, server: hdp-master.mydomain.com/10.200.104.187:2181
2021-05-20 17:51:09,211 - INFO  [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@1273] - Session establishment complete on server hdp-master.mydomain.com/10.200.104.187:2181, sessionid = 0x17988299eff0059, negotiated timeout = 30000
2021-05-20 17:51:19,799 - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.6-315--1, built on 08/23/2019 05:02 GMT
2021-05-20 17:51:19,801 - INFO  [main:Environment@100] - Client environment:host.name=hdp-slave1.mydomain.com
2021-05-20 17:51:19,801 - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_262
2021-05-20 17:51:19,804 - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2021-05-20 17:51:19,804 - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre
2021-05-20 17:51:19,804 - INFO  [main:Environment@100] - Client environment:java.class.path=/usr/hdp/current/zookeeper-client/bin/../build/classes:/usr/hdp/current/zookeeper-client/bin/../build/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/netty-3.10.5.Final.jar:/usr/hdp/current/zookeeper-client/bin/../lib/log4j-1.2.16.jar:/usr/hdp/current/zookeeper-client/bin/../lib/jline-0.9.94.jar:/usr/hdp/current/zookeeper-client/bin/../zookeeper-3.4.6.3.1.4.0-315.jar:/usr/hdp/current/zookeeper-client/bin/../src/java/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../conf::/usr/share/zookeeper/*

 

Explorer

After moving HIVE to another slave node, HIVE is now working but,

If I try to invoke hive shell from slave1 i.e. the problematic host, it tries to connect to the same host i.e. slave1.

I compared 'hive-site.xml' of slave1 & slave2. 
No difference.


Some suspect from the hive-site.xml which might causing slave1 to connect to slave1 after invoking hive shell =

 <property>
      <name>atlas.rest.address</name>
      <value>http://hdp-slave1.mydomain.com:21000</value>
    </property>

<property>
      <name>hive.cluster.delegation.token.store.zookeeper.connectString</name>
      <value>hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181</value>
    </property>

    <property>
      <name>hive.zookeeper.quorum</name>
      <value>hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181</value>
    </property>






 

Mentor

@dmharshit 

Check your Ambari UI--->HDFS-->Config-- look for the parameter hadoop.proxyuser.hive.hosts make sure it's the HS2 host is listed here.

 

ClouderaComm_HDP 3.1.4.JPG

 

Restart the stale config and let me know!

Explorer

@Shelton  Thanks for the reply.

I confirmed that it's HS2 listed in hadoop.proxyuser.hive.hosts i.e. slave2 host in my case which is running Hive server service by now.

Mentor

@dmharshit 
Is the problem still persistent?  Have you tried using Ambari REST API  to move the component and delete the component for the old host?
Please revert

Explorer

Problem is still there.

 

21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1127.19.1.el7.x86_64
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:user.name=eagledev
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/user1
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/user1
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@5ace1ed4
21/06/09 16:50:22 INFO zookeeper.ClientCnxn: Opening socket connection to server hdp-slave1.mydomain.com/10.200.104.188:2181. Will not attempt to authenticate using SASL (unknown error)
21/06/09 16:50:22 INFO zookeeper.ClientCnxn: Socket connection established to hdp-slave1.mydomain.com/10.200.104.188:2181, initiating session
21/06/09 16:50:22 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
21/06/09 16:50:22 INFO zookeeper.ClientCnxn: Session establishment complete on server hdp-slave1.mydomain.com/10.200.104.188:2181, sessionid = 0x279ef5fd2c3006b, negotiated timeout = 60000
21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Session: 0x279ef5fd2c3006b closed
21/06/09 16:50:22 INFO zookeeper.ClientCnxn: EventThread shut down
org.apache.curator.CuratorZookeeperClient.startAdvancedTracer(Ljava/lang/String;)Lorg/apache/curator/drivers/OperationTrace;
Beeline version 3.1.0.3.1.4.0-315 by Apache Hive
0: jdbc:hive2://hdp-slave1.mydomain.com:2 (closed)>