Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Why the number of reducer determined by Hadoop MapReduce and Tez has a great differ?

avatar
Explorer

Hive on tez,sometimes the reduce number of tez is very fewer,in hadoop mapreduce has 2000 reducers, but in tez only 10.This cause take a long time to complete the query task.

the hive.exec.reducers.bytes.per.reducer is same.Is there any mistake in judging the Map output in tez?

how can to solve this problem?

1 ACCEPTED SOLUTION

avatar
Master Mentor
@Jun Chen

Tez architecture is different from mapreduce.

http://hortonworks.com/hadoop/tez/

Tez requires fewer jobs (1 vs. 3) and no IO synchronization barriers (provided via HDFS for the MR jobs) are required.

753-screen-shot-2015-12-10-at-71226-am.png

View solution in original post

6 REPLIES 6

avatar
Master Mentor
@Jun Chen

Tez architecture is different from mapreduce.

http://hortonworks.com/hadoop/tez/

Tez requires fewer jobs (1 vs. 3) and no IO synchronization barriers (provided via HDFS for the MR jobs) are required.

753-screen-shot-2015-12-10-at-71226-am.png

avatar
Rising Star

If you're experiencing performance issue on Tez you need to start checking hive.tez.container.size: we had worked a lot in Hive / Tez performance optimization and very often you need to check your jobs. Sometimes we lowered the hive.tez.container.size to 1024 (less memory means more containers), other times we need to set the property value to 8192. It really depend on your workload.

Hive / Tez optimization could be a real long work but you can achieve good performance using hive.tez.container.size, ORC (and compression algorithm) and "pre-warming" Tez container.

avatar

@Jun Chen check if you have parameter below turned on:

hive.tez.auto.reducer.parallelism

when it's on, tez automatically decrease number of reducer tasks based on output from map. you can disable it if you need.

avatar
Explorer

Tez set very few reduces initially before automatically decreasing.Following is the detail picture:

773-image-1.png

avatar

@Jun Chen

I see... I know tez has a new way to define number of mappers tasks, described in link below, not sure about number of reducers. Usually, we define a high number of reducers by default (in ambari) and use auto.reducer parameter, that works well.

https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works

avatar
Master Mentor

@Jun Chen are you still having issues with this? Can you accept best answer or provide your own solution?