Support Questions
Find answers, ask questions, and share your expertise

Hive metastore wont start after enabling kerberos

Highlighted

Hive metastore wont start after enabling kerberos

Explorer

Hello, I am having 3 node cluster (sandbox) with one master node and 2 data nodes running HDP 2.5.3 .

I have enabled kerberos and since then hive metastore is not able to start, it has the following errors. I have tried different solutions provided by this forum, reboot all 3 nodes and restarted all, but still facing the same errors. Any help please, I am learrning for the first time how to enable kerberos using Ambari. I also checked mysql database used by hive is up and running.

Please help, Thanks,

Wassem

Metastore on master.hadoop.com failed (Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py", line 198, in execute
    timeout=int(check_command_timeout) )
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call
    raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of 'export HIVE_CONF_DIR='/usr/hdp/current/hive-metastore/conf/conf.server' ; hive --hiveconf hive.metastore.uris=thrift://master.hadoop.com:9083                 --hiveconf hive.metastore.client.connect.retry.delay=1                 --hiveconf hive.metastore.failure.retries=1                 --hiveconf hive.metastore.connect.retries=1                 --hiveconf hive.metastore.client.socket.timeout=14                 --hiveconf hive.execution.engine=mr -e 'show databases;'' returned 1. Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/conf.server/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:543)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1551)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:89)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:135)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:107)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3252)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3271)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
	... 8 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1549)
	... 14 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:446)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1549)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:89)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:135)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:107)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3252)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3271)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
	... 30 more
)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:492)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
	... 19 more
)








6 REPLIES 6
Highlighted

Re: Hive metastore wont start after enabling kerberos

@Wissem abdel ben

Based on the log you provided the error seems to indicate is not able to open connection to metastore port due connection refused (usually meaning no process is listening on this port - or firewall is blocking the connection)

I recommend you go to the metastore host and check:

1. the logs under /var/log/hive/hivemetastore.log

2. see if metastore process is up and running by issuing ps -ef |grep metastore

3. double check if another process is listenting in port 9083 netstat -nap |grep 9083

4. if you see metastore process running and listening on port perhaps you should check firewall rules

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Highlighted

Re: Hive metastore wont start after enabling kerberos

Explorer

Thank you,

Here is output of the following commands

[root@master conf]# service mysqld status
mysqld (pid  3216) is running...
[root@master conf]# ps -ef |grep metastore
root      75261   2902  0 08:33 pts/0    00:00:00 grep metastore
[root@master conf]# netstat -nap |grep 9083
[root@master conf]#

also firewall is disabled


[root@master conf]# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination


Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination


Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination


[root@master conf]#


Highlighted

Re: Hive metastore wont start after enabling kerberos

Explorer

79459-hive.png

Above the ambari webconsole

Highlighted

Re: Hive metastore wont start after enabling kerberos

@Wissem abdel ben

So hive metastore is not running, try to start just metastore from ambari while tailing the logs I mentioned above: /var/log/hive/hivemetastore.log

Highlighted

Re: Hive metastore wont start after enabling kerberos

Explorer
[hive@master ~]$  kinit -kt /etc/security/keytabs/hive.service.keytab hive/_HOST@HADOOP.COM
kinit: Keytab contains no suitable keys for hive/_HOST@HADOOP.COM while getting initial credentials
[hive@master ~]$  kinit -kt /etc/security/keytabs/hive.service.keytab hive/master.hadoop.com@HADOOP.COM
[hive@master ~]$ beeline -u 'jdbc:hive2://master.hadoop.com:10000/;transportMode=binary;principal=hive/master.hadoop.com@HADOOP.COM'
Connecting to jdbc:hive2://master.hadoop.com:10000/;transportMode=binary;principal=hive/master.hadoop.com@HADOOP.COM
18/07/12 08:45:25 [main]: WARN jdbc.HiveConnection: Failed to connect to master.hadoop.com:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master.hadoop.com:10000/;transportMode=binary;principal=hive/master.hadoop.com@HADOOP.COM: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Beeline version 1.2.1000.2.5.3.0-37 by Apache Hive
0: jdbc:hive2://master.hadoop.com:10000/ (closed)>


Highlighted

Re: Hive metastore wont start after enabling kerberos

Explorer

hive metastore is not running because it cannot be started via Ambari console, i tried to start it many times but it shutdowns :(