Reply
Explorer
Posts: 58
Registered: ‎09-29-2016

Hive Metastore problem with Oracle DB

[ Edited ]

Hi Folks,

I am migrating the Hive metastore to oracle from derby. i am able to setup all hive metastore tables in oracle  through .hive/bin/schematool -initSchema -dbType oracle. when starting the metaservice its not getting started. i checked the hive log and can see below error. any suggestion.solutions will be much appreciated.

 

2016-09-29 04:07:08,084 WARN [main]: metastore.HiveMetaStore (HiveMetaStore.java:createDefaultDB(671)) - Retrying creating default database after error: Exception thrown while loading remaining rows of query
javax.jdo.JDOUserException: Exception thrown while loading remaining rows of query
at org.datanucleus.api.jdo.JDOAdapter.getUserExceptionForException(JDOAdapter.java:1139)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:296)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:107)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:236)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:755)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:94)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:447)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:500)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:427)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:182)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:543)
at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:7127)
at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:7085)
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7043)
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7027)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy2.verifySchema(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:621)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:669)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:478)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5885)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5880)
at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6132)
at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6057)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
NestedThrowablesStackTrace:
java.lang.NullPointerException
2016-09-29 04:07:37,274 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6225)) - javax.jdo.JDOUserException: Exception thrown while loading remaining rows of query
at org.datanucleus.api.jdo.JDOAdapter.getUserExceptionForException(JDOAdapter.java:1139)

 

 

 

Cloudera Employee
Posts: 34
Registered: ‎08-16-2016

Re: Hive Metastore problem with Oracle DB

Is there additional stack from this exception? if so could you please post it?
Also what version of CDH is this?
Can you also post the value of the following properties in hive-site.xml?
javax.jdo.option.ConnectionURL
javax.jdo.option.ConnectionDriverName
javax.jdo.option.ConnectionUserName
javax.jdo.option.ConnectionPassword

Thanks
Explorer
Posts: 58
Registered: ‎09-29-2016

Re: Hive Metastore problem with Oracle DB

Hi,

I am using /hadoop-2.6.0-cdh5.7.0

these are the properties :

<property>
<name>javax.jdo.option.ConnectionURL</name>
<!-- <value>jdbc:derby://sd-b6d4-b03e/cdh5metastore</value> -->
<!-- <value>jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = abdb.nm.root.net)(PORT = 1521))(CONNECT_DATA = (SID = cfddbuat)))</value> -->
<value>jdbc:oracle:thin:@abdb.nm.root.net:1521:cfddbuat</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<!-- <value>org.apache.derby.jdbc.ClientDriver</value> -->
<value>oracle.jdbc.OracleDriver</value>
<description>Driver class name for a JDBC metastore</description>
</property>

<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>username to use against metastore database</description>
</property>

<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>xxx</value>

 

Also using java 1.8,oracle version : 11.2.0.4 ,driver : ojdbc7.jar

No more logs.only that log i am getting.

 

 

Cloudera Employee
Posts: 34
Registered: ‎08-16-2016

Re: Hive Metastore problem with Oracle DB

Thanks for the info. Those props look right to me.
Also quickly, ensure that the schema version is correct. ( I have no reason to believe it is wrong but just for check it off).
Using Oracle DB client, login to oracle DB, and run select * from VERSION;
the SCHEMA_VERSION should be 1.1.0.

This error is thrown by the datanucleus layer. Its hard to know the exact cause as to what triggers this from the JDO layer.
The following jira, with the same error message, suggests some kinda thread contention in the PMF implementation.
http://www.datanucleus.org/servlet/jira/browse/NUCRDBMS-587

Perhaps this can help with some clues.
Highlighted
Explorer
Posts: 58
Registered: ‎09-29-2016

Re: Hive Metastore problem with Oracle DB

Thanks..I resolved the problem.i downloaded ojdbc6 jar exact version of
Oracle and then it worked.Previously I tried ojdbc6 11.2.0.3 then tried
ojdbc7.
It was problem with jdo only.
Though any version of ojdbc6 should have worked but this is something
don't know the real root cause but it worked.
I think some conflict of ojdbc driver with jdo.