Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

Hive meta store fails to start with MetaException column does not exist

avatar
author-rank jamescross91
Explorer

Created on ‎01-12-2015 06:53 AM - edited ‎09-16-2022 02:18 AM

I see this exception for a range of tables and columns:

 

12:58:10.926 PMERRORorg.apache.hadoop.hive.metastore.ObjectStore
Error retrieving statistics via jdo
MetaException(message:Column TPOW_OPT1TPRICE doesn't exist.)
	at org.apache.hadoop.hive.metastore.ObjectStore.validateTableCols(ObjectStore.java:5741)
	at org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:5692)
	at org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:147)
	at org.apache.hadoop.hive.metastore.ObjectStore$5.getJdoResult(ObjectStore.java:5763)
	at org.apache.hadoop.hive.metastore.ObjectStore$5.getJdoResult(ObjectStore.java:5755)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2223)
	at org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:5755)
	at org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:5749)
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
	at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:3568)
	at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)
	at com.sun.proxy.$Proxy5.get_table_statistics_req(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:9528)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:9512)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

 

 

In addition to this:

 

2:32:56.515 PMERRORorg.apache.hadoop.hive.metastore.RetryingHMSHandler
NoSuchObjectException(message:There is no database named cloudera_manager_metastore_canary_test_db_hive_hivemetastore_be54987370afe9de58b7ef7de50b5db5)
	at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:495)
	at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:506)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
	at com.sun.proxy.$Proxy0.getDatabase(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:805)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)
	at com.sun.proxy.$Proxy5.get_database(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_database.getResult(ThriftHiveMetastore.java:7696)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_database.getResult(ThriftHiveMetastore.java:7680)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

 

Any help or suggestions would be hugely appreciated.

 

James

16,270 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank brock author-rank
Contributor

Created ‎02-22-2015 10:55 AM

I think this might be https://issues.apache.org/jira/browse/HIVE-9620 which is caused for avro tables and a case difference on column names.

View solution in original post

16,375 Views
0 Kudos
1
5 REPLIES 5

avatar
author-rank jamescross91
Explorer

Created ‎01-12-2015 06:58 AM

Forgot to mention, running CDH 5.3, Impala 2.1.0

16,267 Views
0 Kudos

avatar
author-rank brock author-rank
Contributor

Created ‎01-12-2015 09:33 AM

Is TPOW_OPT1TPRICE a column in one of your tables? It's certainly not a column in the HMS. When do you get this error about TPOW_OPT1TPRICE and does it fail the query?

 

Cheers,

Brock

16,260 Views
0 Kudos

avatar
author-rank jamescross91
Explorer

Created ‎01-12-2015 09:41 AM

Brock,

 

Thank you for your reply.  Yes this is one of our columns.  The failure occurs when the hive metastore starts.  We see these stack traces for lots of columns that we have added (possibly even all of them).  If we try and exectue a query again this table we get a broken pipe unable to connect to metastore exception.  Hue also fails when trying to browse metadata.

 

James

16,258 Views
0 Kudos

avatar
author-rank brock author-rank
Contributor

Created ‎02-22-2015 10:55 AM

I think this might be https://issues.apache.org/jira/browse/HIVE-9620 which is caused for avro tables and a case difference on column names.

16,376 Views
0 Kudos
1

avatar
author-rank jamescross91
Explorer

Created ‎02-23-2015 01:53 AM

Hi Brock - thanks for your reply.  I had previously solved the problem, but yes this was the issue.

 

James

16,166 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements