question Re: hiveserver2 not starting in HDP 3.1.4 in Support Questions

hiveserver2 not starting in HDP 3.1.4

dmharshit — Thu, 20 May 2021 07:32:04 GMT

Hello,

I am getting following error when trying to start Hive server =

2021-05-20 12:21:50,088 - Will retry 7 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s) 2021-05-20 12:22:00,098 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server hdp-master.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-slave1.mydomain.com:2181 ls /hiveserver2 | grep 'serverUri=''] {} 2021-05-20 12:22:00,780 - call returned (1, '') 2021-05-20 12:22:00,781 - Will retry 6 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s) 2021-05-20 12:22:10,791 - Process with pid 12962 is not running. Stale pid file at /var/run/hive/hive-server.pid

I referred to similar threads at this forum but didn't help.

Pls suggest how to fix this issue thank you.

Regards,

Amey.

Re: hiveserver2 not starting in HDP 3.1.4

tusharkathpal — Thu, 20 May 2021 11:40:44 GMT

@dmharshit

A znode is supposed to be formed under zookeeper. Check for zookeeper logs during the HS2 start as to why it is or HS2 is not able to create znode.

Re: hiveserver2 not starting in HDP 3.1.4

dmharshit — Thu, 20 May 2021 12:27:40 GMT

@tusharkathpal Thanks.
zookeeper logs aren't showing any useful info I think =

2021-05-20 17:51:09,093 - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp 2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:java.compiler=<NA> 2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:os.name=Linux 2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:os.arch=amd64 2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:os.version=3.10.0-1127.19.1.el7.x86_64 2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:user.name=root 2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:user.home=/root 2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:user.dir=/var/lib/ambari-agent 2021-05-20 17:51:09,095 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=hdp-master.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-slave1.mydomain.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@25f38edc 2021-05-20 17:51:09,120 - INFO [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@1013] - Opening socket connection to server hdp-master.mydomain.com/10.200.104.187:2181. Will not attempt to authenticate using SASL (unknown error) 2021-05-20 17:51:09,190 - INFO [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@856] - Socket connection established, initiating session, client: /10.200.104.188:37684, server: hdp-master.mydomain.com/10.200.104.187:2181 2021-05-20 17:51:09,211 - INFO [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@1273] - Session establishment complete on server hdp-master.mydomain.com/10.200.104.187:2181, sessionid = 0x17988299eff0059, negotiated timeout = 30000 2021-05-20 17:51:19,799 - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-315--1, built on 08/23/2019 05:02 GMT 2021-05-20 17:51:19,801 - INFO [main:Environment@100] - Client environment:host.name=hdp-slave1.mydomain.com 2021-05-20 17:51:19,801 - INFO [main:Environment@100] - Client environment:java.version=1.8.0_262 2021-05-20 17:51:19,804 - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation 2021-05-20 17:51:19,804 - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre 2021-05-20 17:51:19,804 - INFO [main:Environment@100] - Client environment:java.class.path=/usr/hdp/current/zookeeper-client/bin/../build/classes:/usr/hdp/current/zookeeper-client/bin/../build/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/netty-3.10.5.Final.jar:/usr/hdp/current/zookeeper-client/bin/../lib/log4j-1.2.16.jar:/usr/hdp/current/zookeeper-client/bin/../lib/jline-0.9.94.jar:/usr/hdp/current/zookeeper-client/bin/../zookeeper-3.4.6.3.1.4.0-315.jar:/usr/hdp/current/zookeeper-client/bin/../src/java/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../conf::/usr/share/zookeeper/*

Re: hiveserver2 not starting in HDP 3.1.4

dmharshit — Fri, 21 May 2021 07:24:10 GMT

After moving HIVE to another slave node, HIVE is now working but,

If I try to invoke hive shell from slave1 i.e. the problematic host, it tries to connect to the same host i.e. slave1.

I compared 'hive-site.xml' of slave1 & slave2.
No difference.

Some suspect from the hive-site.xml which might causing slave1 to connect to slave1 after invoking hive shell =

<property> <name>atlas.rest.address</name> <value>http://hdp-slave1.mydomain.com:21000</value> </property> <property> <name>hive.cluster.delegation.token.store.zookeeper.connectString</name> <value>hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181</value> </property> <property> <name>hive.zookeeper.quorum</name> <value>hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181</value> </property>

Re: hiveserver2 not starting in HDP 3.1.4

Shelton — Sat, 22 May 2021 20:30:30 GMT

@dmharshit

Check your Ambari UI--->HDFS-->Config-- look for the parameter hadoop.proxyuser.hive.hosts make sure it's the HS2 host is listed here.

Restart the stale config and let me know!

Re: hiveserver2 not starting in HDP 3.1.4

dmharshit — Wed, 02 Jun 2021 10:44:06 GMT

@Shelton Thanks for the reply.

I confirmed that it's HS2 listed in hadoop.proxyuser.hive.hosts i.e. slave2 host in my case which is running Hive server service by now.

Re: hiveserver2 not starting in HDP 3.1.4

Shelton — Thu, 03 Jun 2021 21:17:59 GMT

@dmharshit
Is the problem still persistent? Have you tried using Ambari REST API to move the component and delete the component for the old host?
Please revert

Re: hiveserver2 not starting in HDP 3.1.4

dmharshit — Wed, 09 Jun 2021 11:24:15 GMT

Problem is still there.

21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1127.19.1.el7.x86_64 21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:user.name=eagledev 21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/user1 21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/user1 21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@5ace1ed4 21/06/09 16:50:22 INFO zookeeper.ClientCnxn: Opening socket connection to server hdp-slave1.mydomain.com/10.200.104.188:2181. Will not attempt to authenticate using SASL (unknown error) 21/06/09 16:50:22 INFO zookeeper.ClientCnxn: Socket connection established to hdp-slave1.mydomain.com/10.200.104.188:2181, initiating session 21/06/09 16:50:22 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting 21/06/09 16:50:22 INFO zookeeper.ClientCnxn: Session establishment complete on server hdp-slave1.mydomain.com/10.200.104.188:2181, sessionid = 0x279ef5fd2c3006b, negotiated timeout = 60000 21/06/09 16:50:22 INFO zookeeper.ZooKeeper: Session: 0x279ef5fd2c3006b closed 21/06/09 16:50:22 INFO zookeeper.ClientCnxn: EventThread shut down org.apache.curator.CuratorZookeeperClient.startAdvancedTracer(Ljava/lang/String;)Lorg/apache/curator/drivers/OperationTrace; Beeline version 3.1.0.3.1.4.0-315 by Apache Hive 0: jdbc:hive2://hdp-slave1.mydomain.com:2 (closed)>