question Re: Hive Compaction for ACID Transactions in Archives of Support Questions (Read Only)

Hive Compaction for ACID Transactions

PranayV — Mon, 08 Feb 2016 05:25:54 GMT

Hello Hive SME's,

I am setting up Hive tables for ACID Transactions. There are fair(not high) number of inserts/ updates expected on each tables.

Should the compactions be scheduled every day? or let Hive manage compaction? Are there any pros/cons on hive managed compactions?

Hive Version 0.14

Thank You

Pranay Vyas

Re: Hive Compaction for ACID Transactions

nsabharwal — Mon, 08 Feb 2016 05:34:54 GMT

@pvyas

Please see this before you do this in prod

https://community.hortonworks.com/content/kbentry/4321/hive-acid-current-state.html

Re: Hive Compaction for ACID Transactions

PranayV — Mon, 08 Feb 2016 05:39:39 GMT

Thanks @Neeraj Sabharwal

Re: Hive Compaction for ACID Transactions

ekoifman — Tue, 09 Feb 2016 03:24:37 GMT

How often to run compaction is a function of how quickly you are generating delta files (see https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-BasicDesignfor more info). Less frequent compactions will make reads more expensive. Keep in mind that this system is designed for slowly changing data. Updating 1 row out of 1 billion row table every second will not work well. The cost of executing an SQL UPDATE statement that matches 1 row and 10K rows is roughly the same.

The other response regarding the state of this feature in Hive 0.14 is still valid.

Re: Hive Compaction for ACID Transactions

PranayV — Thu, 11 Feb 2016 00:57:25 GMT

Thank You @Eugene Koifman. this helps.