Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Hive metastore wont start after enabling kerberos

Labels:
wissen_sou
Explorer

Created ‎07-12-2018 02:48 PM

Hello, I am having 3 node cluster (sandbox) with one master node and 2 data nodes running HDP 2.5.3 .

I have enabled kerberos and since then hive metastore is not able to start, it has the following errors. I have tried different solutions provided by this forum, reboot all 3 nodes and restarted all, but still facing the same errors. Any help please, I am learrning for the first time how to enable kerberos using Ambari. I also checked mysql database used by hive is up and running.

Please help, Thanks,

Wassem

Metastore on master.hadoop.com failed (Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py", line 198, in execute
    timeout=int(check_command_timeout) )
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call
    raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of 'export HIVE_CONF_DIR='/usr/hdp/current/hive-metastore/conf/conf.server' ; hive --hiveconf hive.metastore.uris=thrift://master.hadoop.com:9083                 --hiveconf hive.metastore.client.connect.retry.delay=1                 --hiveconf hive.metastore.failure.retries=1                 --hiveconf hive.metastore.connect.retries=1                 --hiveconf hive.metastore.client.socket.timeout=14                 --hiveconf hive.execution.engine=mr -e 'show databases;'' returned 1. Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/conf.server/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:543)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1551)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:89)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:135)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:107)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3252)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3271)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
	... 8 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1549)
	... 14 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:446)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1549)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:89)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:135)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:107)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3252)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3271)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
	... 30 more
)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:492)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
	... 19 more
)
1,534 Views
0 Kudos
6 REPLIES 6

falbani
Guru

Created ‎07-12-2018 03:17 PM

@Wissem abdel ben

Based on the log you provided the error seems to indicate is not able to open connection to metastore port due connection refused (usually meaning no process is listening on this port - or firewall is blocking the connection)

I recommend you go to the metastore host and check:

1. the logs under /var/log/hive/hivemetastore.log

2. see if metastore process is up and running by issuing ps -ef |grep metastore

3. double check if another process is listenting in port 9083 netstat -nap |grep 9083

4. if you see metastore process running and listening on port perhaps you should check firewall rules

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

1,441 Views

wissen_sou
Explorer

Created ‎07-12-2018 03:36 PM

Thank you,

Here is output of the following commands

[root@master conf]# service mysqld status
mysqld (pid  3216) is running...
[root@master conf]# ps -ef |grep metastore
root      75261   2902  0 08:33 pts/0    00:00:00 grep metastore
[root@master conf]# netstat -nap |grep 9083
[root@master conf]#

also firewall is disabled


[root@master conf]# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination


Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination


Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination


[root@master conf]#
1,441 Views
0 Kudos

wissen_sou
Explorer

Created on ‎07-12-2018 03:39 PM - edited ‎08-18-2019 12:33 AM

79459-hive.png

Above the ambari webconsole

1,441 Views
0 Kudos

falbani
Guru

Created ‎07-12-2018 03:47 PM

@Wissem abdel ben

So hive metastore is not running, try to start just metastore from ambari while tailing the logs I mentioned above: /var/log/hive/hivemetastore.log

1,441 Views
0 Kudos

wissen_sou
Explorer

Created ‎07-12-2018 03:47 PM 

[hive@master ~]$  kinit -kt /etc/security/keytabs/hive.service.keytab hive/_HOST@HADOOP.COM
kinit: Keytab contains no suitable keys for hive/_HOST@HADOOP.COM while getting initial credentials
[hive@master ~]$  kinit -kt /etc/security/keytabs/hive.service.keytab hive/master.hadoop.com@HADOOP.COM
[hive@master ~]$ beeline -u 'jdbc:hive2://master.hadoop.com:10000/;transportMode=binary;principal=hive/master.hadoop.com@HADOOP.COM'
Connecting to jdbc:hive2://master.hadoop.com:10000/;transportMode=binary;principal=hive/master.hadoop.com@HADOOP.COM
18/07/12 08:45:25 [main]: WARN jdbc.HiveConnection: Failed to connect to master.hadoop.com:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master.hadoop.com:10000/;transportMode=binary;principal=hive/master.hadoop.com@HADOOP.COM: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Beeline version 1.2.1000.2.5.3.0-37 by Apache Hive
0: jdbc:hive2://master.hadoop.com:10000/ (closed)>
1,441 Views
0 Kudos

wissen_sou
Explorer

Created ‎07-12-2018 04:10 PM

hive metastore is not running because it cannot be started via Ambari console, i tried to start it many times but it shutdowns 😞

1,441 Views
0 Kudos
Take a Tour of the Community
Don't have an account?
Register
Your experience may be limited. Sign in to explore more.
Announcements
Community Announcements
March 2023 Community Highlights
What's New @ Cloudera
Cloudera DataFlow Designer for self-service data flow development is now generally available to all CDP Public Cloud customers
What's New @ Cloudera
Cloudera Operational Database (COD) UI provides the JWT configuration details to connect to your HBase client
What's New @ Cloudera
Cloudera Operational Database (COD) UI allows storage type selection when creating an operational database
What's New @ Cloudera
Release of Cloudera Streaming Analytics (CSA) 1.9.0 for CDP 7.1.7 SP2 & CDP 7.1.8
View More Announcements