Support Questions

Find answers, ask questions, and share your expertise

hiveserver2 and hive metastore fail to start (HDP 2.6.3.0)

Explorer

Checked through the various other like questions. No solution found there.... Of course I very new to ambari.

I am unable to get hiveserver2 and hive metastore to start. It is unsure if hive is using mysql or postgress. There may have been a typo in the have db password. I've followed a number of hints to set the hive user (hive) password in the mysql db users table.

Where to start? (hah! I've been looking into this for 2 days.)

These services are running on a single machine:

Activity Analyzer Activity Explorer App Timeline Server HDFS Client History Server Hive Client Hive Metastore HiveServer2 HST Agent HST Server MapReduce2 Client Metrics Collector Grafana Metrics Monitor MySQL Server Pig Client ResourceManager Slider Client Spark2 History Server Spark Client Spark History Server Tez Client WebHCat Server YARN Client ZooKeeper Client ZooKeeper Server

6 REPLIES 6

Explorer

What error do you see in the logs for hiveserver2/metastore?

From Ambari you can go to Hive-> Configs and you should be able to see the DB that is set and credentials as well.

Explorer

I've been tryong lots of things. I've tried to move hive and its metastore to different servers....

Hive Metastore Process (RetryingMetaStoreClient.java:92) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3526) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3558) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:533) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564) ... 14 more Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused) at org.apache.thrift.transport.TSocket.open(TSocket.java:226) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:487) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:282) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:76) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:92) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3526) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3558) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:533) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.thrift.transport.TSocket.open(TSocket.java:221) ... 22 more ) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:534) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:282) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:76) ... 19 more )" data-bindattr-7396="7396" style="font-size: 0.8em; width: 575.703px; overflow-x: auto; overflow-y: hidden; text-overflow: inherit; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; line-height: 20px; background-color: rgb(255, 255, 255);">Metastore on ambari.cslu.ohsu.edu failed (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py", line 203, in execute timeout_kill_strategy=TerminateStrategy.KILL_PROCESS_TREE, File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/u...



HiveServer2 Process &1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'' returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://ambari.cslu.ohsu.edu:10000/;transportMode=binary: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0) Error: Could not open client transport with JDBC Uri: jdbc:hive2://ambari.cslu.ohsu.edu:10000/;transportMode=binary: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0) )" data-bindattr-7423="7423" style="font-size: 0.8em; width: 575.703px; overflow-x: auto; overflow-y: hidden; text-overflow: inherit; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; line-height: 20px; background-color: rgb(255, 255, 255);">Connection failed on host ambari.cslu.ohsu.edu:10000 (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py", line 211, in execute ldap_password=ldap_password) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/hive_check.py", line 81, in check_thrift_port_sasl timeout...

Explorer

I'm still digging into this....

Hive check gives:

Message: Cannot open a hive connection with connect string jdbc:hive2://hadoopns1.cslu.ohsu.edu:2181,hadoopns2.cslu.ohsu.edu:2181,ambari.cslu.ohsu.edu:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;hive.server2.proxy.user=admin

Why does the Hiveserver2 JDBC URL list hadoopns1, hadoopns2 on the url when hive runs on the ambari machine?

ns1 and ns2 are running:

  • HST Agent
  • Metrics Monitor
  • NameNode
  • ZooKeeper Server

hadoopserv01 is running:

  • HST Agent
  • MapReduce2 Client
  • Metrics Monitor
  • Tez Client
  • YARN Client
  • ZooKeeper Server

The ambari machine is running 26 components including:

  • Hive Client
  • Hive Metastore
  • HiveServer2

On a side note I am able to log directly into mysql as hive with the expected password.

Explorer

It may be that the hive database was not created successfully during the install. When testing the hive database connection I get this error:

stderr:
2018-01-08 16:21:35,419 - Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties.
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'hive'

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 530, in <module>
    CheckHost().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 207, in actionexecute
    raise Fail(error_message)
resource_management.core.exceptions.Fail: Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties.
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'hive'
stdout:
2018-01-08 16:21:34,958 - Host checks started.
2018-01-08 16:21:34,958 - Check execute list: db_connection_check
2018-01-08 16:21:34,958 - DB connection check started.
WARNING: File /var/lib/ambari-agent/cache/DBConnectionVerification.jar already exists, assuming it was downloaded before

2018-01-08 16:21:34,968 - call['/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-agent/cache/DBConnectionVerification.jar:/var/lib/ambari-agent/cache/mysql-connector-java.jar -Djava.library.path=/var/lib/ambari-agent/cache org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://ambari.cslu.ohsu.edu/hive" "hive" [PROTECTED] com.mysql.jdbc.Driver'] {}
2018-01-08 16:21:35,418 - call returned (1, "ERROR: Unable to connect to the DB. Please check DB connection properties.\ncom.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'hive'")
2018-01-08 16:21:35,418 - DB connection check completed.
2018-01-08 16:21:35,419 - Host checks completed.
2018-01-08 16:21:35,419 - Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties.
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'hive'

Command failed after 1 tries

If that is the case how do I reinstall hive?

Super Mentor

@ethan vanmatre

the following error indicates that the Hive Services are not setup / installed correctly.

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:Unknown database 'hive'

.

So Pleas uninstall the Hive Service as following:

1. Login to Ambari UI as "admin" use and then navigate to:

Ambari UI --> Hive --> Service Actions (Drop Down at right corner) --> Stop

2. Now Delete the service as following:

Ambari UI --> Hive --> Service Actions (Drop Down at right corner) --> "Delete Service"

.

Once the Hive Service is removed successfully then please try to install it again from ambari UI as :

Ambari UI --> "Actions" (Drop Down) --> "Add Service"  --> Hive

.

If you are planning to use a Non Default Database for HIve then please refer to the prerequisite steps hee:

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-administration/content/using_hive...

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-administration/content/using_non-...

.

Explorer

Thanks, That did the trick - as suspected there was an incomplete install of the hive database (mysql). Removing hive required that spark and spark2 were also removed and waiting for the system to remove all of the clients. TTT (things take time) so wait for it.

The reinstall of hive, spark and spark2 went well and hive is now running. Again TTT and thanks.

--Ethan

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.