hive metastore server fails to start with postgresql db error

hadoopranger — Fri, 31 May 2024 14:39:36 GMT

We have changed our postgresql DB instance to a new port and migrated all the locally available data from older DB instance to a newer one

Updated the port every place needed in cloudera manager and db.properties

However while starting the hive metastore server is failing with below error

P.S : We are using postgres as external database

ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [TThreadPoolServer WorkerProcess-20]: Retrying HMSHandler after 2000 ms (attempt 3 of 10) with error: javax.jdo.JDOUserException: Exception thrown while loading remaining rows of query
at org.datanucleus.api.jdo.JDOAdapter.getUserExceptionForException(JDOAdapter.java:659)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:310)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
NestedThrowablesStackTrace:
Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
org.datanucleus.exceptions.NucleusDataStoreException: Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:147)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceNonLoadedFields(StateManagerImpl.java:3166)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchNonLoadedFields(PersistentClassROF.java:491)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3114)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.processNumberOfResults(ForwardQueryResult.java:143)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.advanceToEndOfResultSet(ForwardQueryResult.java:171)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:298)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
... 60 more
Caused by: org.postgresql.util.PSQLException: ERROR: large object 57710 does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
at org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
at org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
at org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1416)
... 32 more
Nested Throwables StackTrace:
org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceNonLoadedFields(StateManagerImpl.java:3166)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchNonLoadedFields(PersistentClassROF.java:491)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3114)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.processNumberOfResults(ForwardQueryResult.java:143)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.advanceToEndOfResultSet(ForwardQueryResult.java:171)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:298)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.postgresql.util.PSQLException: ERROR: large object 57710 does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
at org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
at org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
at org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1416)
... 32 more

Re: hive metastore server fails to start with postgresql db error

smruti — Sat, 01 Jun 2024 06:05:33 GMT

@hadoopranger It seems like a transaction is in failed state. a drop database call probably did not complete successfully. Did we make config changes in postgres while Hive service was running? That could lead to data corruption.
I notice the following error:

Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block org.datanucleus.exceptions.NucleusDataStoreException: Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block

You may get the out of the following command and see which transaction is in idle but aborted state.

select * from pg_stat_activity;

Then try to terminate the transaction or perform a ROLLBACK. Take a backup of the database before making these changes to be on the safer side.

question Re: hive metastore server fails to start with postgresql db error in Support Questions

hive metastore server fails to start with postgresql db error

Re: hive metastore server fails to start with postgresql db error

Re: hive metastore server fails to start with postgresql db error