Skip to main content
dmharshit
Silver ⭐️⭐️⭐️
May 20, 2021 7:32 AM UTC
Question

hiveserver2 not starting in HDP 3.1.4

Hello,

I am getting following error when trying to start Hive server =

 

2021-05-20 12:21:50,088 - Will retry 7 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2021-05-20 12:22:00,098 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server hdp-master.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-slave1.mydomain.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2021-05-20 12:22:00,780 - call returned (1, '')
2021-05-20 12:22:00,781 - Will retry 6 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2021-05-20 12:22:10,791 - Process with pid 12962 is not running. Stale pid file at /var/run/hive/hive-server.pid

 

 

I referred to similar threads at this forum but didn't help.

Pls suggest how to fix this issue thank you.

Regards,

Amey.

7 replies

tusharkathpal
Gold ⭐️
Gold ⭐️
May 20, 2021 11:40 AM UTC

@dmharshit 

 

A znode is supposed to be formed under zookeeper. Check for zookeeper logs during the HS2 start as to why it is or HS2 is not able to create znode.

 

Translate
dmharshit
dmharshitAuthor
Silver ⭐️⭐️⭐️
May 20, 2021 12:27 PM UTC

@tusharkathpal  Thanks.
zookeeper logs aren't showing any useful info I think =

2021-05-20 17:51:09,093 - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:os.name=Linux
2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:os.arch=amd64
2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:os.version=3.10.0-1127.19.1.el7.x86_64
2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:user.name=root
2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:user.home=/root
2021-05-20 17:51:09,094 - INFO [main:Environment@100] - Client environment:user.dir=/var/lib/ambari-agent
2021-05-20 17:51:09,095 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=hdp-master.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-slave1.mydomain.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@25f38edc
2021-05-20 17:51:09,120 - INFO [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@1013] - Opening socket connection to server hdp-master.mydomain.com/10.200.104.187:2181. Will not attempt to authenticate using SASL (unknown error)
2021-05-20 17:51:09,190 - INFO [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@856] - Socket connection established, initiating session, client: /10.200.104.188:37684, server: hdp-master.mydomain.com/10.200.104.187:2181
2021-05-20 17:51:09,211 - INFO [main-SendThread(hdp-master.mydomain.com:2181):ClientCnxn$SendThread@1273] - Session establishment complete on server hdp-master.mydomain.com/10.200.104.187:2181, sessionid = 0x17988299eff0059, negotiated timeout = 30000
2021-05-20 17:51:19,799 - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-315--1, built on 08/23/2019 05:02 GMT
2021-05-20 17:51:19,801 - INFO [main:Environment@100] - Client environment:host.name=hdp-slave1.mydomain.com
2021-05-20 17:51:19,801 - INFO [main:Environment@100] - Client environment:java.version=1.8.0_262
2021-05-20 17:51:19,804 - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2021-05-20 17:51:19,804 - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre
2021-05-20 17:51:19,804 - INFO [main:Environment@100] - Client environment:java.class.path=/usr/hdp/current/zookeeper-client/bin/../build/classes:/usr/hdp/current/zookeeper-client/bin/../build/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/netty-3.10.5.Final.jar:/usr/hdp/current/zookeeper-client/bin/../lib/log4j-1.2.16.jar:/usr/hdp/current/zookeeper-client/bin/../lib/jline-0.9.94.jar:/usr/hdp/current/zookeeper-client/bin/../zookeeper-3.4.6.3.1.4.0-315.jar:/usr/hdp/current/zookeeper-client/bin/../src/java/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../conf::/usr/share/zookeeper/*

 

Translate
dmharshit
dmharshitAuthor
Silver ⭐️⭐️⭐️
May 21, 2021 7:24 AM UTC

After moving HIVE to another slave node, HIVE is now working but,

If I try to invoke hive shell from slave1 i.e. the problematic host, it tries to connect to the same host i.e. slave1.

I compared 'hive-site.xml' of slave1 & slave2. 
No difference.


Some suspect from the hive-site.xml which might causing slave1 to connect to slave1 after invoking hive shell =

 <property>
<name>atlas.rest.address</name>
<value>http://hdp-slave1.mydomain.com:21000</value>
</property>

<property>
<name>hive.cluster.delegation.token.store.zookeeper.connectString</name>
<value>hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181</value>
</property>

<property>
<name>hive.zookeeper.quorum</name>
<value>hdp-slave1.mydomain.com:2181,hdp-slave2.mydomain.com:2181,hdp-master.mydomain.com:2181</value>
</property>






 

Translate
Shelton
Gold ⭐️⭐️⭐️
May 22, 2021 8:30 PM UTC

@dmharshit 

Check your Ambari UI--->HDFS-->Config-- look for the parameter hadoop.proxyuser.hive.hosts make sure it's the HS2 host is listed here.

 

ClouderaComm_HDP 3.1.4.JPG

 

Restart the stale config and let me know!

Translate
dmharshit
dmharshitAuthor
Silver ⭐️⭐️⭐️
Jun 2, 2021 10:44 AM UTC

@Shelton  Thanks for the reply.

I confirmed that it's HS2 listed in hadoop.proxyuser.hive.hosts i.e. slave2 host in my case which is running Hive server service by now.

Translate
Shelton
Gold ⭐️⭐️⭐️
Jun 3, 2021 9:17 PM UTC

@dmharshit 
Is the problem still persistent?  Have you tried using Ambari REST API  to move the component and delete the component for the old host?
Please revert

Translate