Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive : Can we enable concurrency support without enabling ACID ?

avatar

My use case is such that there are concurrent jobs which end up firing INSERT INTO queries to same table.

As such my namenode logs are getting filled with HDFS StateChange WARN logs.

I do not want to enable ACID as it does not support External Tables.

Can we only enable concurrency support by setting following properties while still keeping ACID disabled ?

hive.support.concurrency = true
hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
hive.enforce.bucketing = true
hive.exec.dynamic.partition.mode =  nonstrict

The default/current values are

hive.support.concurrency = false
hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
hive.enforce.bucketing = true
hive.exec.dynamic.partition.mode = nonstrict
1 ACCEPTED SOLUTION

avatar
Super Collaborator
  1. hive.support.concurrency =true
  2. hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
  3. hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager

will install a lock manager (there are several; ZooKeeper based is the default) w/o enabling full Acid.

If you do use

hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager then hive.lock.manager is ignored and you will be using Metastore based lock manager that is used by Acid but if you don't create your tables with "transactional=true" all your tables remain the same. I believe external tables should be locked in this case.

View solution in original post

2 REPLIES 2

avatar

@Sindhu - Would you be able to help with this ? TIA

avatar
Super Collaborator
  1. hive.support.concurrency =true
  2. hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
  3. hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager

will install a lock manager (there are several; ZooKeeper based is the default) w/o enabling full Acid.

If you do use

hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager then hive.lock.manager is ignored and you will be using Metastore based lock manager that is used by Acid but if you don't create your tables with "transactional=true" all your tables remain the same. I believe external tables should be locked in this case.