question Hive : Can we enable concurrency support without enabling ACID ? in Archives of Support Questions (Read Only)

Hive : Can we enable concurrency support without enabling ACID ?

dineshc — Fri, 29 Sep 2017 10:02:59 GMT

My use case is such that there are concurrent jobs which end up firing INSERT INTO queries to same table.

As such my namenode logs are getting filled with HDFS StateChange WARN logs.

I do not want to enable ACID as it does not support External Tables.

Can we only enable concurrency support by setting following properties while still keeping ACID disabled ?

hive.support.concurrency = true
hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
hive.enforce.bucketing = true
hive.exec.dynamic.partition.mode =  nonstrict

The default/current values are

hive.support.concurrency = false
hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
hive.enforce.bucketing = true
hive.exec.dynamic.partition.mode = nonstrict

Re: Hive : Can we enable concurrency support without enabling ACID ?

dineshc — Fri, 29 Sep 2017 10:03:49 GMT

@Sindhu - Would you be able to help with this ? TIA

Re: Hive : Can we enable concurrency support without enabling ACID ?

ekoifman — Fri, 29 Sep 2017 23:07:59 GMT

hive.support.concurrency =true
hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager

will install a lock manager (there are several; ZooKeeper based is the default) w/o enabling full Acid.

If you do use

hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager then hive.lock.manager is ignored and you will be using Metastore based lock manager that is used by Acid but if you don't create your tables with "transactional=true" all your tables remain the same. I believe external tables should be locked in this case.