Tez with Transaction with Bucketing

[#1]

When a table is partitioned and bucketed and Transactions enabled on it , the number of map tasks launched by TEZ = 2 , while MR jobs still launches 72 Tasks (Table is about 17Gig). if transaction is not enabled , then the query is launching Correct number of Tez tasks, If there are any hints on why this may occur, please share.

[#2]

@[email protected]

What version of Hive and HDP are you using?

[#3]

Are the Tez jobs submitting to the same queue as MR jobs? (hive.server2.tez.default.queues, hive.server2.tez.sessions.per.default.queue)

How do Tez container settings compare with general YARN container settings?(tez.am.resource.memory.mb, tez.am.java.opts, hive.tez.container.size, hive.tez.java.opts)

[#4]

@[email protected], @Ryan Templeton and @[email protected] found same issue. I think @ravi is interacting with engineering in order to fix or workaround it.

[#5 · Accepted solution]

After compacting (Major compaction per partition) the tables (HDP 2.2.4.2-2) we got the right number of Tez mappers. So this appears to be a bug related to compaction.

Alex,

Yes, they are both submitting to the same queue.

The ACID Transactions is broken until further advisary. I am looking for more details of the reasons why it is broken. If you have details please send me a note.

[#6]

May be related to https://issues.apache.org/jira/browse/HIVE-9977

[#7]

A few things to note:

• If the data was inserted into the table using Hive streaming there will be lot of small files, these get compacted into large files only on compaction. If you haven't enabled compactor in the metastore then they will not be compacted, in that case you will need to issue a compaction explicitly.
• Recommendation for customers on HDP 2.2 is to not deploy Transactions in production, there are issues with bucketing going awry, meaning you could end up with potential data corruption. There will be fixes for these in the upcoming HDP 2.3 releases so stay tuned.

Bottomline, we should refrain from deploying Transactions in HDP 2.2 releases.