hiveserver2 not starting (Zookeeper not ready)

[#1]

I have a newly installed hdp 3.0.0 on a single cluster.

Everything starts without issue, except hiveserver2.

Logs are saying that hiveserver2 is timing out due to a failure of zookeeper not finding the /hiveserver2 node.

I can connect to the zookeeper client and when I list out the node it is empty ( [] ).

Here's an excerpt from the relevant part of the logs:

2018-08-30 11:51:35,220 - call['ambari-sudo.sh su hive -l -s /bin/bash -c 'hive --config /usr/hdp/current/hive-server2/conf/ --service metatool -listFSRoot' 2>/dev/null | grep hdfs:// | cut -f1,2,3 -d '/' | grep -v 'hdfs://mnhost3.corp.pitechsol.com:8020' | head -1'] {}
2018-08-30 11:51:56,704 - call returned (0, '')
2018-08-30 11:51:56,704 - Execute['/var/lib/ambari-agent/tmp/start_hiveserver2_script /var/log/hive/hive-server2.out /var/log/hive/hive-server2.err /var/run/hive/hive-server.pid /usr/hdp/current/hive-server2/conf/ /etc/tez/conf'] {'environment': {'HIVE_BIN': 'hive', 'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_112', 'HADOOP_HOME': u'/usr/hdp/current/hadoop-client'}, 'not_if': 'ls /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps -p  >/dev/null 2>&1', 'user': 'hive', 'path': [u'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hive-server2/bin:/usr/hdp/3.0.0.0-1634/hadoop/bin']}
2018-08-30 11:51:56,799 - Execute['/usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-server2/lib/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://mnhost3.corp.pitechsol.com/hive?createDatabaseIfNotExist=true&useSSL=false&allowPublicKeyRetrieval=true' hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 'try_sleep': 10}
2018-08-30 11:51:57,499 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server mnhost3.corp.pitechsol.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2018-08-30 11:51:58,720 - call returned (1, '')
2018-08-30 11:51:58,721 - Will retry 29 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2018-08-30 11:52:08,732 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server mnhost3.corp.pitechsol.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2018-08-30 11:52:09,601 - call returned (1, '')
2018-08-30 11:52:09,612 - Will retry 28 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2018-08-30 11:52:19,623 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server mnhost3.corp.pitechsol.com:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2018-08-30 11:52:20,815 - call returned (1, '')

I guess the real question I have is why is the /hiveserver2 node not getting initialized within zookeeper? Where/When/What/How is the the hiveserver2 node created/initialized?

I've made no changes from the default parameters on the install. This is on a RHEL 7.5 OS if that matters any.

[#2]

Hi @Michael north,

Not sure if its same,are you hitting : https://issues.apache.org/jira/browse/AMBARI-23943

have you configured ViewFS ? : https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/managing-and-monitoring-ambari/content/amb_...

[#3]

I didn't do anything that would lead me to believe I configured viewFS into the system.

I am using the stock configuration as downloaded from Hortonworks for HDP 3.0.0

Michael

[#4]

@Michael North

The znode is created when you fire up zk it read and uploads the config of most of the components. I would think you need to revise your hive setup.

Switch directory to /usr/hdp/current/zookeeper-client/

$ ./bin/zkCli.sh 

Could you check your ZK namespace under the hiveserver2 path you should have an entry like below

[zk: localhost:2181(CONNECTED) o] ls /hiveserver2 
[serverUri=nakuru.kenya.ke:10000;version=1.2.1000.2.6.2.0-205;sequence=0000000061] 
[zk: localhost:2181(CONNECTED) 1] 

Can you check if HS2 can reach 2181 port of ZK host which in your case is the same machine? Have you configured a metastore for your hive either MySQL, Postgres or otherwise? if not you need to do that See my attached screenshot When the HS2 is running and only then when you restart will zk pick the configuration and the znode as shown above

HTH

[#5]

Hello @Shelton,

I have this same issue.

I configured the metastore for your hive either MySQL.

But the below command returned []

ls /hiveserver2 

Could you please help me with this.

@north Do you recal how you configured this.

[#6]

I am getting the same output when performing command

ls /hiveserver2

Any possible solution acquired?

[#7]

@Shelton 

Tried the commands.

Output =

[zk: localhost:2181(CONNECTED) 0] ls /hiveserver2
[]
[zk: localhost:2181(CONNECTED) 1]

Metastore for Hive is MySQL in my case.

Can you pls suggest further steps to fix the issue? Thanks.
Hive is unable to start on my HDP 3.1.4 cluster.

Regards,

Amey.

[#8]

-Try the below.

-When you start the HS2, track the logs and look for the first error in HS2.

-Parallelly ensure that your zookeeper is up and running and check the zk logs during the restart to validate if the connections are coming to ZK and whether zk is accepting or rejecting the connection.

[#9]

@dmharshit 

Check your Ambari UI--->HDFS-->Config-- look for the parameter hadoop.proxyuser.hive.hosts make sure it's the HS2 host is listed here.

In my case bern.swiss.ch

Then restart the stale services and revert if you see any errors share the logs

Happy Hadooping

[#10]

mine fix by ... Install Ranger

please check your stderr ....