Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

hive metastore server fails to start with postgresql db error

Labels:
avatar
author-rank hadoopranger
Rising Star

Created ‎05-31-2024 07:39 AM

We have changed our postgresql DB instance to a new port and migrated all the locally available data from older DB instance to a newer one

Updated the port every place needed in cloudera manager and db.properties

However while starting the hive metastore server is failing with below error

P.S : We are using postgres as external database 

ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [TThreadPoolServer WorkerProcess-20]: Retrying HMSHandler after 2000 ms (attempt 3 of 10) with error: javax.jdo.JDOUserException: Exception thrown while loading remaining rows of query
at org.datanucleus.api.jdo.JDOAdapter.getUserExceptionForException(JDOAdapter.java:659)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:310)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
NestedThrowablesStackTrace:
Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
org.datanucleus.exceptions.NucleusDataStoreException: Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:147)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceNonLoadedFields(StateManagerImpl.java:3166)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchNonLoadedFields(PersistentClassROF.java:491)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3114)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.processNumberOfResults(ForwardQueryResult.java:143)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.advanceToEndOfResultSet(ForwardQueryResult.java:171)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:298)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
... 60 more
Caused by: org.postgresql.util.PSQLException: ERROR: large object 57710 does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
at org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
at org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
at org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1416)
... 32 more
Nested Throwables StackTrace:
org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceNonLoadedFields(StateManagerImpl.java:3166)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchNonLoadedFields(PersistentClassROF.java:491)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3114)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.processNumberOfResults(ForwardQueryResult.java:143)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.advanceToEndOfResultSet(ForwardQueryResult.java:171)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:298)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.postgresql.util.PSQLException: ERROR: large object 57710 does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
at org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
at org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
at org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1416)
... 32 more

2,403 Views
2 REPLIES 2

avatar
author-rank smruti author-rank
Master Collaborator

Created ‎05-31-2024 11:05 PM

@hadoopranger It seems like a transaction is in failed state. a drop database call probably did not complete successfully. Did we make config changes in postgres while Hive service was running? That could lead to data corruption. 
I notice the following error:

Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
org.datanucleus.exceptions.NucleusDataStoreException: Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block

You may get the out of the following command and see which transaction is in idle but aborted state.  

select * from pg_stat_activity;

Then try to terminate the transaction or perform a ROLLBACK. Take a backup of the database before making these changes to be on the safer side.

2,386 Views

avatar
author-rank hadoopranger
Rising Star

Created on ‎06-01-2024 12:49 AM - edited ‎06-01-2024 04:53 AM

There has been a update in external db hive is using and new version instance is running on a new port this config change was made when whole cluster was stopped

@smruti Can you please  help with how is the data loaded from hive metastore to postgres sql is it through pyscopg and where is it stored in database server

cause we need to get the missing data from older version of db but cannot locate where the source of this corrupted data is on db server

2,380 Views
Announcements
Community Announcements
Community Upgrade Announcement
What's New @ Cloudera
Announcing Cloudera Flow Management Operator for Kubernetes ...
What's New @ Cloudera
Announcing Cloudera Streams Messaging Operator for Kubernete...
Community Announcements
July 2026 Community Highlights
Community Announcements
June 2026 Community Highlights
View More Announcements