- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
hive metastore server fails to start with postgresql db error
- Labels:
-
Apache Hive
Created ‎05-31-2024 07:39 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We have changed our postgresql DB instance to a new port and migrated all the locally available data from older DB instance to a newer one
Updated the port every place needed in cloudera manager and db.properties
However while starting the hive metastore server is failing with below error
P.S : We are using postgres as external database
ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [TThreadPoolServer WorkerProcess-20]: Retrying HMSHandler after 2000 ms (attempt 3 of 10) with error: javax.jdo.JDOUserException: Exception thrown while loading remaining rows of query
at org.datanucleus.api.jdo.JDOAdapter.getUserExceptionForException(JDOAdapter.java:659)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:310)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
NestedThrowablesStackTrace:
Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
org.datanucleus.exceptions.NucleusDataStoreException: Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:147)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceNonLoadedFields(StateManagerImpl.java:3166)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchNonLoadedFields(PersistentClassROF.java:491)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3114)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.processNumberOfResults(ForwardQueryResult.java:143)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.advanceToEndOfResultSet(ForwardQueryResult.java:171)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:298)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
... 60 more
Caused by: org.postgresql.util.PSQLException: ERROR: large object 57710 does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
at org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
at org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
at org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1416)
... 32 more
Nested Throwables StackTrace:
org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceNonLoadedFields(StateManagerImpl.java:3166)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchNonLoadedFields(PersistentClassROF.java:491)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3114)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.processNumberOfResults(ForwardQueryResult.java:143)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.advanceToEndOfResultSet(ForwardQueryResult.java:171)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.closingConnection(ForwardQueryResult.java:298)
at org.datanucleus.store.query.AbstractQueryResult.disconnect(AbstractQueryResult.java:105)
at org.datanucleus.store.rdbms.query.AbstractRDBMSQueryResult.disconnect(AbstractRDBMSQueryResult.java:248)
at org.datanucleus.store.rdbms.query.JDOQLQuery$2.transactionPreClose(JDOQLQuery.java:714)
at org.datanucleus.store.connection.AbstractManagedConnection.transactionPreClose(AbstractManagedConnection.java:101)
at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionPreRollBack(ConnectionManagerImpl.java:445)
at org.datanucleus.TransactionImpl.internalPreRollback(TransactionImpl.java:506)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:433)
at org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:739)
at org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:12056)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.isPartOfMaterializedView(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:3032)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3324)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:3311)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1908)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:2014)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.drop_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16455)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:16434)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.postgresql.util.PSQLException: ERROR: large object 57710 does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)
at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:817)
at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:604)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:110)
at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:152)
at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:164)
at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:107)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:246)
at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:229)
at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:272)
at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:117)
at org.postgresql.jdbc.PgClob.getCharacterStream(PgClob.java:54)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getObject(ClobRDBMSMapping.java:124)
at org.datanucleus.store.rdbms.mapping.datastore.ClobRDBMSMapping.getString(ClobRDBMSMapping.java:107)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:185)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:142)
at org.datanucleus.state.AbstractStateManager.replacingStringField(AbstractStateManager.java:1575)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:120)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceField(MCreationMetadata.java)
at org.apache.hadoop.hive.metastore.model.MCreationMetadata.dnReplaceFields(MCreationMetadata.java)
at org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
at org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
at org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
at org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at org.apache.hadoop.hive.metastore.ObjectStore.isPartOfMaterializedView(ObjectStore.java:1416)
... 32 more
Created ‎05-31-2024 11:05 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@hadoopranger It seems like a transaction is in failed state. a drop database call probably did not complete successfully. Did we make config changes in postgres while Hive service was running? That could lead to data corruption.
I notice the following error:
Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
org.datanucleus.exceptions.NucleusDataStoreException: Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block
You may get the out of the following command and see which transaction is in idle but aborted state.
select * from pg_stat_activity;
Then try to terminate the transaction or perform a ROLLBACK. Take a backup of the database before making these changes to be on the safer side.
Created on ‎06-01-2024 12:49 AM - edited ‎06-01-2024 04:53 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There has been a update in external db hive is using and new version instance is running on a new port this config change was made when whole cluster was stopped
@smruti Can you please help with how is the data loaded from hive metastore to postgres sql is it through pyscopg and where is it stored in database server
cause we need to get the missing data from older version of db but cannot locate where the source of this corrupted data is on db server
