Hive meta store fails to start with MetaException column does not exist

jamescross91 — Fri, 16 Sep 2022 09:18:37 GMT

I see this exception for a range of tables and columns:

12:58:10.926 PM

ERROR

org.apache.hadoop.hive.metastore.ObjectStore

Error retrieving statistics via jdo
MetaException(message:Column TPOW_OPT1TPRICE doesn't exist.)
	at org.apache.hadoop.hive.metastore.ObjectStore.validateTableCols(ObjectStore.java:5741)
	at org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:5692)
	at org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:147)
	at org.apache.hadoop.hive.metastore.ObjectStore$5.getJdoResult(ObjectStore.java:5763)
	at org.apache.hadoop.hive.metastore.ObjectStore$5.getJdoResult(ObjectStore.java:5755)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2223)
	at org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:5755)
	at org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:5749)
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
	at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:3568)
	at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)
	at com.sun.proxy.$Proxy5.get_table_statistics_req(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:9528)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:9512)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

In addition to this:

2:32:56.515 PM

ERROR

org.apache.hadoop.hive.metastore.RetryingHMSHandler

NoSuchObjectException(message:There is no database named cloudera_manager_metastore_canary_test_db_hive_hivemetastore_be54987370afe9de58b7ef7de50b5db5)
	at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:495)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:506)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
	at com.sun.proxy.$Proxy0.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:805)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)
	at com.sun.proxy.$Proxy5.get_database(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_database.getResult(ThriftHiveMetastore.java:7696)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_database.getResult(ThriftHiveMetastore.java:7680)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Any help or suggestions would be hugely appreciated.

James

Re: Hive meta store fails to start with MetaException column does not exist

jamescross91 — Mon, 12 Jan 2015 14:58:30 GMT

Forgot to mention, running CDH 5.3, Impala 2.1.0

Re: Hive meta store fails to start with MetaException column does not exist

brock — Mon, 12 Jan 2015 17:33:41 GMT

Is TPOW_OPT1TPRICE a column in one of your tables? It's certainly not a column in the HMS. When do you get this error about TPOW_OPT1TPRICE and does it fail the query?

Cheers,

Brock

Re: Hive meta store fails to start with MetaException column does not exist

jamescross91 — Mon, 12 Jan 2015 17:41:27 GMT

Brock,

Thank you for your reply. Yes this is one of our columns. The failure occurs when the hive metastore starts. We see these stack traces for lots of columns that we have added (possibly even all of them). If we try and exectue a query again this table we get a broken pipe unable to connect to metastore exception. Hue also fails when trying to browse metadata.

James

Re: Hive meta store fails to start with MetaException column does not exist

brock — Sun, 22 Feb 2015 18:55:59 GMT

I think this might be https://issues.apache.org/jira/browse/HIVE-9620 which is caused for avro tables and a case difference on column names.

Re: Hive meta store fails to start with MetaException column does not exist

jamescross91 — Mon, 23 Feb 2015 09:53:17 GMT

Hi Brock - thanks for your reply. I had previously solved the problem, but yes this was the issue.

James

question Re: Hive meta store fails to start with MetaException column does not exist in Archives of Support Questions (Read Only)

Hive meta store fails to start with MetaException column does not exist

Re: Hive meta store fails to start with MetaException column does not exist

Re: Hive meta store fails to start with MetaException column does not exist

Re: Hive meta store fails to start with MetaException column does not exist

Re: Hive meta store fails to start with MetaException column does not exist

Re: Hive meta store fails to start with MetaException column does not exist