About ekoifman

ekoifman · ‎01-16-2017

They system doesn't think your table is an Acid table. Can you post "describe formatted XYZ"? (Note that above you have an extra " in DDL) do you have hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager? Are you able to "select from XYZ"?

ekoifman · ‎12-07-2016

These tables are created by insert statement using VALUES clause - it's part of internal implementation logic. They are temp tables and are session scoped and as you've noticed will go away when the session ends.

ekoifman · ‎10-04-2016

Assuming 'id' is never null in either table select case when s.id is null then t.id else s.id end, case when s.id is null then t.count else s.count end from old_data t full outer join new_data s on t.id = s.id though what you really want is the MERGE statement which is WIP (https://issues.apache.org/jira/browse/HIVE-10924)

ekoifman · ‎08-22-2016

If you are doing "set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager" at CLI it won't always work. You have to set this option in hive-site.xml (and restart) to make sure (it's a bug that's been fixed in HIVE-11716). You also have to make sure that "set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; set hive.compactor.initiator.on=true; set hive.compactor.worker.threads=1;" are set for your standalone metastore process process which is required for ACID.

ekoifman · ‎08-10-2016

https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration has a list of configuration options. For example, hive.compactor.delta.num.threshold, lets you control how often minor compaction runs wrt number of SQL operations. Each SQL insert/update/delete generates 1 delta file. Each minor compaction, combines whatever delta files it finds into a single new delta that includes all the info. There is not specific guidance available since depends on specific set up/requirements and has to tested for. There other options listed there that can help.

ekoifman · ‎07-15-2016

The relevant bug is https://issues.apache.org/jira/browse/HIVE-13821. Triggering compaction is the right solution.

ekoifman · ‎06-17-2016

both compactions are important. The should run minor compactions more frequently to combine delta files into fewer deltas. From the listing above major compaction ran and produced the base file. All the delta files in the listing should be garbage collected by the system. Could you provide output of "show locks" and "show transactions"? The only thing that should prevent deletion of these delta files is a query that is still running which the "show" commands should indicate.

ekoifman · ‎06-17-2016

is drill client configured for Acid? If you see the change in Hive, you must have set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager and hive.support.concurrency=true. Are the same properties set for the session through which Drill connects to Hive?

ekoifman · ‎06-13-2016

The parallelism of the write side is equal to the number of buckets for the table. For the read side, there was (is) a bug with split computation on Tez. If you do "dfs -lsr" on the table directory, do you see any files called base_xxxxxx? If not, you may want to run a Major compaction. It will generate a base_xxxx file and then your subsequent Updates should have > 1 split. (this was fixed in https://issues.apache.org/jira/browse/HIVE-13821 and requires Tez 0.7) Alternatively, run the update statement on MR - it doesn't have split calculation issue.

ekoifman · ‎06-06-2016

Your Compactor Initiator seems to be running. I don't see any errors. 2016-06-05 09:58:26,907 INFO [Thread-8]: compactor.Initiator (Initiator.java:run(89)) - Checking to see if we should compact default.journey_entidad_acid What version of Hive is this?

Online	Offline
Last Visited	‎01-02-2019 10:23 PM

Member Since	‎12-09-2015 05:12 PM
Last Visited	‎01-02-2019 10:23 PM
Posts	106
Kudos received	40

Cloudera Community

Re: Hive 3 export single ORC file

Re: Hive Compaction error

Re: What is difference between Distributed by,clus...

Re: Hive Transactional Tables are not readable by ...

Re: Updating the bucketted Hive table

Re: ACID Transactions DELETE & UPDATE Issues ... S...

Re: Hive: Temporary table created automatically

Re: insert or update if exists in hive

Re: How to update hive table

Re: HIVE and ACID table performance for updates

Re: Hive ACID table and low performance : select w...

Re: bucket y delta in hive

Re: update hive something strange

Re: hive updates very slow

Re: bucket y delta in hive