Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Regarding to activate HIVE ACID transactions on the cluster

Solved Go to solution

Regarding to activate HIVE ACID transactions on the cluster

New Contributor

Hi all, I need some help here.

Regarding to activate HIVE ACID transactions on the cluster.

Nowadays, we have currently 300-node hadoop cluster - HDP, version 2.6.4.

We've been studying possible issues, collateral effects if we apply/enable this feature globally in our cluster, our legacy hive tables (about 1700) hasn't this feature and we'd like to know if have some issue/damage when enable this on the existing hive tables.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Regarding to activate HIVE ACID transactions on the cluster

Hi @Leonardo Araujo. Enabling ACID will not automatically convert existing tables to transactional tables. You will need to manually (or script) new CREATE TABLE statements with the TRANSACTION property and then insert data into those tables. Be aware of existing requirement for transactional tables such as ORC format and bucketing. In HDP 3.0, all Hive managed tables are ACID by default.

3 REPLIES 3
Highlighted

Re: Regarding to activate HIVE ACID transactions on the cluster

Hi @Leonardo Araujo. Enabling ACID will not automatically convert existing tables to transactional tables. You will need to manually (or script) new CREATE TABLE statements with the TRANSACTION property and then insert data into those tables. Be aware of existing requirement for transactional tables such as ORC format and bucketing. In HDP 3.0, all Hive managed tables are ACID by default.

Re: Regarding to activate HIVE ACID transactions on the cluster

New Contributor

Thank you so much @Scott Shaw,

Best regards, Leo

Re: Regarding to activate HIVE ACID transactions on the cluster

New Contributor

Hi @Leonardo Araujo, don't know if you did it.

About the collateral effect on 2.6.4, there is one to take in account, the Hive Acid activation change the transactional manager from :

hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager

To

hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager

The big impact is the way of Hive manage the locks even for no transactionnal tables as describe in Hive documentation about LockManager .

This could dramatically increase processing times of your jobs using Hive.

You could use the parameter hive.txn.strict.locking.mode to manage the behavior of the LockManager even if the documentation said that it is only available since Hive 2.2, use set -v to show presence of this parameter in Hive 1.2.1000 .

You could hit 2 bugs too :

HIVE-17900 : Compaction of ACID table might fail if table is partitioned by more than one column

HIVE-18293 : Compaction of ACID table failed when using doas

Hope this will help.

Best regards, Eric.


Don't have an account?
Coming from Hortonworks? Activate your account here