Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hiveserver2 not coming up

avatar
New Contributor
exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:578)
    at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:518)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
    at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
    at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
    at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
    at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:545)
    ... 9 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1627)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3317)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3356)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3336)
    at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3590)
    at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
    at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1625)
    ... 23 more
Caused by: javax.jdo.JDOUserException: The query returned more than one instance BUT either unique is set to true or only aggregates are to be returned, so should have returned one result maximum
NestedThrowables:
org.datanucleus.store.query.QueryNotUniqueException: The query returned more than one instance BUT either unique is set to true or only aggregates are to be returned, so should have returned one result maximum
    at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:636)
    at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:388)
    at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:225)
    at org.apache.hadoop.hive.metastore.ObjectStore.getMRole(ObjectStore.java:4091)
    at org.apache.hadoop.hive.metastore.ObjectStore.addRole(ObjectStore.java:3731)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
    at com.sun.proxy.$Proxy21.addRole(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultRoles_core(HiveMetaStore.java:644)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultRoles(HiveMetaStore.java:636)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:399)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6396)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236)
    at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
    ... 28 more
Caused by: org.datanucleus.store.query.QueryNotUniqueException: The query returned more than one instance BUT either unique is set to true or only aggregates are to be returned, so should have returned one result maximum
    at org.datanucleus.store.query.Query.executeQuery(Query.java:1879)
    at org.datanucleus.store.query.Query.executeWithArray(Query.java:1733)
    at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:365)
    ... 45 more
1 ACCEPTED SOLUTION

avatar
Moderator

Hello @sh1vam , 

 

thank you for reporting that your HiveServer2 is not  coming up and throwing the below exception: exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:578)

...

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226) registerAllFunctionsOnce

...

Caused by: javax.jdo.JDOUserException:

...

org.datanucleus.store.query.QueryNotUniqueException: The query returned more than one instance BUT either unique is set to true or only aggregates are to be returned, so should have returned one result maximum at 

...

at org.apache.hadoop.hive.metastore.ObjectStore.getMRole(ObjectStore.java:4091)  

 

I found that your issue is resolved now.  Therefore I would like to add here the strategy that might be helpful for someone to troubleshoot similar issues:

  • Start reading the stacktrace from bottom-up.
  • Read the last "Caused by:" section first. We can see "QueryNotUniqueException".
  • Now start reading the lines after the Caused by to see if you can find a some meaningful classname that is coming from hive. The first one is: org.apache.hadoop.hive.metastore.ObjectStore.getMRole(ObjectStore.java:4091)
  • Google "org.apache.hadoop.hive.metastore.ObjectStore.getMRole" for the source code. The very first match was this one, which was OK for our purpose.
  • The number in brackets is the line number that we should look up. Since we do not know the exact version you are using, we just search for the term getMRole in the hope we will get the clue what it is. Based on the context it seems to be a Metastore Role that we are trying to fetch. So far we know that a metastore role is not unique.
  • Now we start reading further the caused by sentences and find that this caused that the SessionHiveMetaStoreClient could not be instantiated.  

 

Based on your report you found out that the metastore.ROLES table had two admin roles and this was causing the issue and you were using the below commands for your troubleshooting:

mysql> select * from metastore.ROLES;

You identified that the role with id 3 is not required, hence deleted the role:

mysql> delete from metastore.ROLES where ROLE_ID=3;

 

In production environment, please do not forget to create a backup or to have a means of recovery before issuing delete command.

 

Kind regards:

Ferenc


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

View solution in original post

1 REPLY 1

avatar
Moderator

Hello @sh1vam , 

 

thank you for reporting that your HiveServer2 is not  coming up and throwing the below exception: exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:578)

...

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226) registerAllFunctionsOnce

...

Caused by: javax.jdo.JDOUserException:

...

org.datanucleus.store.query.QueryNotUniqueException: The query returned more than one instance BUT either unique is set to true or only aggregates are to be returned, so should have returned one result maximum at 

...

at org.apache.hadoop.hive.metastore.ObjectStore.getMRole(ObjectStore.java:4091)  

 

I found that your issue is resolved now.  Therefore I would like to add here the strategy that might be helpful for someone to troubleshoot similar issues:

  • Start reading the stacktrace from bottom-up.
  • Read the last "Caused by:" section first. We can see "QueryNotUniqueException".
  • Now start reading the lines after the Caused by to see if you can find a some meaningful classname that is coming from hive. The first one is: org.apache.hadoop.hive.metastore.ObjectStore.getMRole(ObjectStore.java:4091)
  • Google "org.apache.hadoop.hive.metastore.ObjectStore.getMRole" for the source code. The very first match was this one, which was OK for our purpose.
  • The number in brackets is the line number that we should look up. Since we do not know the exact version you are using, we just search for the term getMRole in the hope we will get the clue what it is. Based on the context it seems to be a Metastore Role that we are trying to fetch. So far we know that a metastore role is not unique.
  • Now we start reading further the caused by sentences and find that this caused that the SessionHiveMetaStoreClient could not be instantiated.  

 

Based on your report you found out that the metastore.ROLES table had two admin roles and this was causing the issue and you were using the below commands for your troubleshooting:

mysql> select * from metastore.ROLES;

You identified that the role with id 3 is not required, hence deleted the role:

mysql> delete from metastore.ROLES where ROLE_ID=3;

 

In production environment, please do not forget to create a backup or to have a means of recovery before issuing delete command.

 

Kind regards:

Ferenc


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community: