Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive Metastore problem with Oracle DB

Highlighted

Hive Metastore problem with Oracle DB

Hi Folks,

I am migrating the Hive metastore to oracle from derby. i am able to setup all hive metastore tables in oracle  through .hive/bin/schematool -initSchema -dbType oracle. when starting the metaservice its not getting started. i checked the hive log and can see below error. any suggestion.solutions will be much appreciated.

 

2016-09-29 04:07:08,084 WARN [main]: metastore.HiveMetaStore (HiveMetaStore.java:createDefaultDB(671)) - Retrying creating default database after error: Exception thrown while loading remaining rows of query
javax.jdo.JDOUserException: Exception thrown while loading remaining rows of query
at org.datanucleus.api.jdo.JDOAdapter.getUserExceptionForException(JDOAdapter.java:1139)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:296)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:107)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:236)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:755)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:94)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:447)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:500)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:427)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:182)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:543)
at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:7127)
at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:7085)
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7043)
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7027)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy2.verifySchema(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:621)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:669)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:478)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5885)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5880)
at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6132)
at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6057)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
NestedThrowablesStackTrace:
java.lang.NullPointerException
2016-09-29 04:07:37,274 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6225)) - javax.jdo.JDOUserException: Exception thrown while loading remaining rows of query
at org.datanucleus.api.jdo.JDOAdapter.getUserExceptionForException(JDOAdapter.java:1139)

 

 

 

4 REPLIES 4

Re: Hive Metastore problem with Oracle DB

Contributor
Is there additional stack from this exception? if so could you please post it?
Also what version of CDH is this?
Can you also post the value of the following properties in hive-site.xml?
javax.jdo.option.ConnectionURL
javax.jdo.option.ConnectionDriverName
javax.jdo.option.ConnectionUserName
javax.jdo.option.ConnectionPassword

Thanks

Re: Hive Metastore problem with Oracle DB

Hi,

I am using /hadoop-2.6.0-cdh5.7.0

these are the properties :

<property>
<name>javax.jdo.option.ConnectionURL</name>
<!-- <value>jdbc:derby://sd-b6d4-b03e/cdh5metastore</value> -->
<!-- <value>jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = abdb.nm.root.net)(PORT = 1521))(CONNECT_DATA = (SID = cfddbuat)))</value> -->
<value>jdbc:oracle:thin:@abdb.nm.root.net:1521:cfddbuat</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<!-- <value>org.apache.derby.jdbc.ClientDriver</value> -->
<value>oracle.jdbc.OracleDriver</value>
<description>Driver class name for a JDBC metastore</description>
</property>

<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>username to use against metastore database</description>
</property>

<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>xxx</value>

 

Also using java 1.8,oracle version : 11.2.0.4 ,driver : ojdbc7.jar

No more logs.only that log i am getting.

 

 

Re: Hive Metastore problem with Oracle DB

Contributor
Thanks for the info. Those props look right to me.
Also quickly, ensure that the schema version is correct. ( I have no reason to believe it is wrong but just for check it off).
Using Oracle DB client, login to oracle DB, and run select * from VERSION;
the SCHEMA_VERSION should be 1.1.0.

This error is thrown by the datanucleus layer. Its hard to know the exact cause as to what triggers this from the JDO layer.
The following jira, with the same error message, suggests some kinda thread contention in the PMF implementation.
http://www.datanucleus.org/servlet/jira/browse/NUCRDBMS-587

Perhaps this can help with some clues.

Re: Hive Metastore problem with Oracle DB

Thanks..I resolved the problem.i downloaded ojdbc6 jar exact version of
Oracle and then it worked.Previously I tried ojdbc6 11.2.0.3 then tried
ojdbc7.
It was problem with jdo only.
Though any version of ojdbc6 should have worked but this is something
don't know the real root cause but it worked.
I think some conflict of ojdbc driver with jdo.