Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive Metastore issue on HDP 2.5.3.0

Hive Metastore issue on HDP 2.5.3.0

New Contributor

Hi All,

I just installed HDP 2.5.3.0 and Hive Metastore doesn't start. I checked the user in the mysql DB and also I initialized the Hive Data base but still nothing.

I added also the grant for the hive user and I checked the entry in the zookeeper .

Please find below the error message

Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/conf.server/hive-log4j.properties

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:543)

at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.util.RunJar.run(RunJar.java:233)

at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1551)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:89)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:135)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:107)

at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3252)

at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3271)

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)

... 8 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1549)

... 14 more

Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

at org.apache.thrift.transport.TSocket.open(TSocket.java:226)

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:446)

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)

at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1549)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:89)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:135)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:107)

at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3252)

at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3271)

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)

at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.util.RunJar.run(RunJar.java:233)

at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at org.apache.thrift.transport.TSocket.open(TSocket.java:221)

... 22 more

)

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:492)

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)

at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)

... 19 more

Please could some one help me for this case.

Thanks in advance

17 REPLIES 17

Re: Hive Metastore issue on HDP 2.5.3.0

@Avani alamut

Looks like your HiveMetastore process is not running, I can see connection refused from the logs

Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185)

Or there might be some connection issues with hive client and hive metastore machine.

Can you please check connectivity as well ?

From hive client (from where you are running hive command) check if you are able to run below command

telnet <hive-metastore-hostname/ip-address> 9083

On metastore:

try running below command to check is metstore process is running

ps aux|grep HiveMetaStore

try running below command to ensure that metastore process is listening on 9083

root@sandbox ~]# netstat -tulpn|grep 9083
tcp        0      0 0.0.0.0:9083                0.0.0.0:*                   LISTEN      2115/java

Let me know if this helps.

Re: Hive Metastore issue on HDP 2.5.3.0

New Contributor

Hello @nshelke

Thanks a lot for your feedback.

Please find below the responses

# telnet myhostname 9083 
Trying 192.168.100.19... 
telnet: connect to address 192.168.100.19: Connection refused 
#ps aux|grep HiveMetaStore
 No Process 
#netstat -tulpn|grep 9083 
 Nothing 

The Hive Metastore doesn't start, it's why there is no process, but I didn't find how to solve this issue

Thanks again for your help

Re: Hive Metastore issue on HDP 2.5.3.0

@Avani alamut

You can go to Ambari UI and start the Metastore process. If not running,

If you are not using Ambari, use below mathod then,

hive -metastore start

Let me know if this helps.

Re: Hive Metastore issue on HDP 2.5.3.0

New Contributor

Hello @nshelke

The issue is that Metastore is not running, I'm getting the error below

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 259, in <module>
    HiveMetastore().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 285, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 59, in start
    self.configure(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 73, in configure
    hive(name = 'metastore')
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 252, in hive
    mode=0644)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/xml_config.py", line 66, in action_create
    encoding = self.resource.encoding
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create
    content = self._get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content
    return content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 51, in __call__
    return self.get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 142, in get_content
    rendered = self.template.render(self.context)
  File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)
  File "<template>", line 2, in top-level template code
  File "/usr/lib/python2.6/site-packages/ambari_jinja2/filters.py", line 176, in do_dictsort
    return sorted(value.items(), key=sort_func)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
    raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'hivemetastore-site' was not found in configurations dictionary!

Thank you

@nshelke

Highlighted

Re: Hive Metastore issue on HDP 2.5.3.0

Re: Hive Metastore issue on HDP 2.5.3.0

@Avani alamut

What is your ambari-server version?

Can you run below commands and provide output?

1) ls -lrt /usr/share/java/mysql-connector-java*

2) find / -name mysql-connector-java.jar

Re: Hive Metastore issue on HDP 2.5.3.0

New Contributor

Hello @nshelke

I saw the link, they didn't solve the issue, I send then in case of.

Thank you for your help

Re: Hive Metastore issue on HDP 2.5.3.0

New Contributor

Hello @Divakar Annapureddy

What is your ambari-server version? : ambari-2.4.3.0-centos7

Can you run below commands and provide output?

1) ls -lrt /usr/share/java/mysql-connector-java* : I can find it

2) find / -name mysql-connector-java.jar :

/usr/share/java/mysql-connector-java.jar

/usr/hdp/2.5.3.0-37/sqoop/lib/mysql-connector-java.jar

Thank you

Re: Hive Metastore issue on HDP 2.5.3.0

@Avani alamut

It seems like there is some configuration parameter which is invalid. Check for "hivemetastore-site" under Ambari -> Hive configs.