Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive : Can we enable concurrency support without enabling ACID ?

Solved Go to solution
Highlighted

Hive : Can we enable concurrency support without enabling ACID ?

My use case is such that there are concurrent jobs which end up firing INSERT INTO queries to same table.

As such my namenode logs are getting filled with HDFS StateChange WARN logs.

I do not want to enable ACID as it does not support External Tables.

Can we only enable concurrency support by setting following properties while still keeping ACID disabled ?

hive.support.concurrency = true
hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
hive.enforce.bucketing = true
hive.exec.dynamic.partition.mode =  nonstrict

The default/current values are

hive.support.concurrency = false
hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
hive.enforce.bucketing = true
hive.exec.dynamic.partition.mode = nonstrict
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Hive : Can we enable concurrency support without enabling ACID ?

Expert Contributor
  1. hive.support.concurrency =true
  2. hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
  3. hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager

will install a lock manager (there are several; ZooKeeper based is the default) w/o enabling full Acid.

If you do use

hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager then hive.lock.manager is ignored and you will be using Metastore based lock manager that is used by Acid but if you don't create your tables with "transactional=true" all your tables remain the same. I believe external tables should be locked in this case.

View solution in original post

2 REPLIES 2
Highlighted

Re: Hive : Can we enable concurrency support without enabling ACID ?

@Sindhu - Would you be able to help with this ? TIA

Highlighted

Re: Hive : Can we enable concurrency support without enabling ACID ?

Expert Contributor
  1. hive.support.concurrency =true
  2. hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
  3. hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager

will install a lock manager (there are several; ZooKeeper based is the default) w/o enabling full Acid.

If you do use

hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager then hive.lock.manager is ignored and you will be using Metastore based lock manager that is used by Acid but if you don't create your tables with "transactional=true" all your tables remain the same. I believe external tables should be locked in this case.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here