Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

hive metastore server fails to start with postgresql db error

Labels:
avatar
author-rank hadoopranger
Rising Star

Created ‎05-31-2024 07:39 AM

We have changed our postgresql DB instance to a new port and migrated all the locally available data from older DB instance to a newer one

Updated the port every place needed in cloudera manager and db.properties

However while starting the hive metastore server is failing with below error

P.S : We are using postgres as external database 

ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [TThreadPoolServer WorkerProcess-20]: Retrying HMSHandler after 2000 ms (attempt 3 of 10) with error: javax.jdo.JDOUserException: Exception thrown while loading remaining rows of query
at org.datanucleus.api.jdo.JDOAdapter.getUserExceptionForException(JDOAdapter.java:659)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:310)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
NestedThrowablesStackTrace:
Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
org.datanucleus.exceptions.NucleusDataStoreException: Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:147)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceNonLoadedFields(StateManagerImpl.java:3166)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchNonLoadedFields(PersistentClassROF.java:491)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3114)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.processNumberOfResults(ForwardQueryResult.java:143)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.advanceToEndOfResultSet(ForwardQueryResult.java:171)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:298)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
... 60 more
Caused by: org.postgresql.util.PSQLException: ERROR: large object 57710 does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
at org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
at org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
at org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1416)
... 32 more
Nested Throwables StackTrace:
org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceNonLoadedFields(StateManagerImpl.java:3166)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchNonLoadedFields(PersistentClassROF.java:491)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3114)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.processNumberOfResults(ForwardQueryResult.java:143)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.advanceToEndOfResultSet(ForwardQueryResult.java:171)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:298)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.postgresql.util.PSQLException: ERROR: large object 57710 does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
at org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
at org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
at org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1416)
... 32 more

702 Views
2 REPLIES 2

avatar
author-rank smruti author-rank
Master Collaborator

Created ‎05-31-2024 11:05 PM

@hadoopranger It seems like a transaction is in failed state. a drop database call probably did not complete successfully. Did we make config changes in postgres while Hive service was running? That could lead to data corruption. 
I notice the following error:

Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
org.datanucleus.exceptions.NucleusDataStoreException: Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block

You may get the out of the following command and see which transaction is in idle but aborted state.  

select * from pg_stat_activity;

Then try to terminate the transaction or perform a ROLLBACK. Take a backup of the database before making these changes to be on the safer side.

685 Views

avatar
author-rank hadoopranger
Rising Star

Created on ‎06-01-2024 12:49 AM - edited ‎06-01-2024 04:53 AM

There has been a update in external db hive is using and new version instance is running on a new port this config change was made when whole cluster was stopped

@smruti Can you please  help with how is the data loaded from hive metastore to postgres sql is it through pyscopg and where is it stored in database server

cause we need to get the missing data from older version of db but cannot locate where the source of this corrupted data is on db server

679 Views
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements