Support Questions
Find answers, ask questions, and share your expertise

hiveserver2 not starting (Zookeeper not ready)

hiveserver2 not starting (Zookeeper not ready)

New Contributor

I have a newly installed hdp 3.0.0 on a single cluster.

Everything starts without issue, except hiveserver2.

Logs are saying that hiveserver2 is timing out due to a failure of zookeeper not finding the /hiveserver2 node.

I can connect to the zookeeper client and when I list out the node it is empty ( [] ).

Here's an excerpt from the relevant part of the logs:

2018-08-30 11:51:35,220 - call['ambari-sudo.sh su hive -l -s /bin/bash -c 'hive --config /usr/hdp/current/hive-server2/conf/ --service metatool -listFSRoot' 2>/dev/null | grep hdfs:// | cut -f1,2,3 -d '/' | grep -v 'hdfs://mnhost3.corp.pitechsol.com:8020' | head -1'] {}
2018-08-30 11:51:56,704 - call returned (0, '')
2018-08-30 11:51:56,704 - Execute['/var/lib/ambari-agent/tmp/start_hiveserver2_script /var/log/hive/hive-server2.out /var/log/hive/hive-server2.err /var/run/hive/hive-server.pid /usr/hdp/current/hive-server2/conf/ /etc/tez/conf'] {'environment': {'HIVE_BIN': 'hive', 'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_112', 'HADOOP_HOME': u'/usr/hdp/current/hadoop-client'}, 'not_if': 'ls /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps -p  >/dev/null 2>&1', 'user': 'hive', 'path': [u'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hive-server2/bin:/usr/hdp/3.0.0.0-1634/hadoop/bin']}
2018-08-30 11:51:56,799 - Execute['/usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-server2/lib/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://mnhost3.corp.pitechsol.com/hive?createDatabaseIfNotExist=true&useSSL=false&allowPublicKeyRetrieval=true' hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 'try_sleep': 10}
2018-08-30 11:51:57,499 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server mnhost3.corp.pitechsol.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2018-08-30 11:51:58,720 - call returned (1, '')
2018-08-30 11:51:58,721 - Will retry 29 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2018-08-30 11:52:08,732 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server mnhost3.corp.pitechsol.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2018-08-30 11:52:09,601 - call returned (1, '')
2018-08-30 11:52:09,612 - Will retry 28 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2018-08-30 11:52:19,623 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server mnhost3.corp.pitechsol.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2018-08-30 11:52:20,815 - call returned (1, '')

I guess the real question I have is why is the /hiveserver2 node not getting initialized within zookeeper? Where/When/What/How is the the hiveserver2 node created/initialized?

I've made no changes from the default parameters on the install. This is on a RHEL 7.5 OS if that matters any.

6 REPLIES 6
Highlighted

Re: hiveserver2 not starting (Zookeeper not ready)

Highlighted

Re: hiveserver2 not starting (Zookeeper not ready)

New Contributor

I didn't do anything that would lead me to believe I configured viewFS into the system.

I am using the stock configuration as downloaded from Hortonworks for HDP 3.0.0

Michael

Highlighted

Re: hiveserver2 not starting (Zookeeper not ready)

Mentor

@Michael North

The znode is created when you fire up zk it read and uploads the config of most of the components. I would think you need to revise your hive setup.

Switch directory to /usr/hdp/current/zookeeper-client/

$ ./bin/zkCli.sh 

Could you check your ZK namespace under the hiveserver2 path you should have an entry like below

[zk: localhost:2181(CONNECTED) o] ls /hiveserver2 
[serverUri=nakuru.kenya.ke:10000;version=1.2.1000.2.6.2.0-205;sequence=0000000061] 
[zk: localhost:2181(CONNECTED) 1] 

Can you check if HS2 can reach 2181 port of ZK host which in your case is the same machine? Have you configured a metastore for your hive either MySQL, Postgres or otherwise? if not you need to do that See my attached screenshot When the HS2 is running and only then when you restart will zk pick the configuration and the znode as shown above

HTH


capture10b.jpg
Highlighted

Re: hiveserver2 not starting (Zookeeper not ready)

New Contributor

Hello @Shelton,

I have this same issue.

I configured the metastore for your hive either MySQL.

 

But the below command returned []

ls /hiveserver2 

 

Could you please help me with this.

@north Do you recal how you configured this.

Highlighted

Re: hiveserver2 not starting (Zookeeper not ready)

New Contributor

I am getting the same output when performing command

ls /hiveserver2

Any possible solution acquired?

Highlighted

Re: hiveserver2 not starting (Zookeeper not ready)

New Contributor

mine fix by ... Install Ranger

please check your stderr ....