Regarding to activate HIVE ACID transactions on the cluster

[#1]

Hi all, I need some help here.

Regarding to activate HIVE ACID transactions on the cluster.

Nowadays, we have currently 300-node hadoop cluster - HDP, version 2.6.4.

We've been studying possible issues, collateral effects if we apply/enable this feature globally in our cluster, our legacy hive tables (about 1700) hasn't this feature and we'd like to know if have some issue/damage when enable this on the existing hive tables.

[#2 · Accepted solution]

Hi @Leonardo Araujo. Enabling ACID will not automatically convert existing tables to transactional tables. You will need to manually (or script) new CREATE TABLE statements with the TRANSACTION property and then insert data into those tables. Be aware of existing requirement for transactional tables such as ORC format and bucketing. In HDP 3.0, all Hive managed tables are ACID by default.

[#3]

Thank you so much @Scott Shaw,

Best regards, Leo

[#4]

Hi @Leonardo Araujo, don't know if you did it.

About the collateral effect on 2.6.4, there is one to take in account, the Hive Acid activation change the transactional manager from :

hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager

To

hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager

The big impact is the way of Hive manage the locks even for no transactionnal tables as describe in Hive documentation about LockManager .

This could dramatically increase processing times of your jobs using Hive.

You could use the parameter hive.txn.strict.locking.mode to manage the behavior of the LockManager even if the documentation said that it is only available since Hive 2.2, use set -v to show presence of this parameter in Hive 1.2.1000 .

You could hit 2 bugs too :

HIVE-17900 : Compaction of ACID table might fail if table is partitioned by more than one column

HIVE-18293 : Compaction of ACID table failed when using doas

Hope this will help.

Best regards, Eric.