Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive ACID table compaction : error log in metastore

Hive ACID table compaction : error log in metastore

New Contributor

Hi,

My sandbox (HDP2.5) shows errors in hive metastore logs :

2016-08-09 10:54:20,068 ERROR Unable to select next element for compaction, Unknown column 'cq_tblproperties' in 'field list'
2016-08-09 10:54:20,069 ERROR Caught an exception in the main loop of compactor worker sandbox.hortonworks.com-33, MetaException(message:Unable to connect to transaction database com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'cq_tblproperties' in 'field list' at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.Util.getInstance(Util.java:386) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1052) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569) at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524) at com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464) at org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:174) at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:86) ) at org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:216) at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:86)

I expect it is not good for ACID table compactions. Might it be a regression since HDP 2.4 ?

Regards

3 REPLIES 3

Re: Hive ACID table compaction : error log in metastore

It seems update is not happen to data store that is why column(cq_tblproperties) is missing.

can you try adding below property and check if it helps.

<property>
 <name>datanucleus.fixedDatastore</name>
 <value>true</value> 
</property>

Re: Hive ACID table compaction : error log in metastore

New Contributor

Thank you for your answer but that property is already set, as you can see below :

6479-datanucleus.png

Highlighted

Re: Hive ACID table compaction : error log in metastore

Expert Contributor

Can you do a 'describe table COMPACTION_QUEUE' in the MySQL backing the metastore? That is the table that should have 'cq_tblproperties'. And yes, Compactions won't run because of this error.

Also please include "select * from VERSION" from the same DB.

Don't have an account?
Coming from Hortonworks? Activate your account here