question Why the number of reducer determined by Hadoop MapReduce and Tez has a great differ? in Archives of Support Questions (Read Only)

Why the number of reducer determined by Hadoop MapReduce and Tez has a great differ?

connectchen — Thu, 10 Dec 2015 14:59:09 GMT

Hive on tez,sometimes the reduce number of tez is very fewer,in hadoop mapreduce has 2000 reducers, but in tez only 10.This cause take a long time to complete the query task.

the hive.exec.reducers.bytes.per.reducer is same.Is there any mistake in judging the Map output in tez?

how can to solve this problem?

Re: Why the number of reducer determined by Hadoop MapReduce and Tez has a great differ?

nsabharwal — Mon, 19 Aug 2019 12:40:11 GMT

@Jun Chen

Tez architecture is different from mapreduce.

http://hortonworks.com/hadoop/tez/

Tez requires fewer jobs (1 vs. 3) and no IO synchronization barriers (provided via HDFS for the MR jobs) are required.

Re: Why the number of reducer determined by Hadoop MapReduce and Tez has a great differ?

dorio — Fri, 11 Dec 2015 05:25:18 GMT

If you're experiencing performance issue on Tez you need to start checking hive.tez.container.size: we had worked a lot in Hive / Tez performance optimization and very often you need to check your jobs. Sometimes we lowered the hive.tez.container.size to 1024 (less memory means more containers), other times we need to set the property value to 8192. It really depend on your workload.

Hive / Tez optimization could be a real long work but you can achieve good performance using hive.tez.container.size, ORC (and compression algorithm) and "pre-warming" Tez container.

Re: Why the number of reducer determined by Hadoop MapReduce and Tez has a great differ?

gbraccialli3 — Fri, 11 Dec 2015 10:20:30 GMT

@Jun Chen check if you have parameter below turned on:

hive.tez.auto.reducer.parallelism

when it's on, tez automatically decrease number of reducer tasks based on output from map. you can disable it if you need.

Re: Why the number of reducer determined by Hadoop MapReduce and Tez has a great differ?

connectchen — Mon, 19 Aug 2019 12:40:03 GMT

Tez set very few reduces initially before automatically decreasing.Following is the detail picture:

Re: Why the number of reducer determined by Hadoop MapReduce and Tez has a great differ?

gbraccialli3 — Fri, 11 Dec 2015 11:15:11 GMT

@Jun Chen

I see... I know tez has a new way to define number of mappers tasks, described in link below, not sure about number of reducers. Usually, we define a high number of reducers by default (in ambari) and use auto.reducer parameter, that works well.

https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works

Re: Why the number of reducer determined by Hadoop MapReduce and Tez has a great differ?

aervits — Wed, 03 Feb 2016 23:46:39 GMT

@Jun Chen are you still having issues with this? Can you accept best answer or provide your own solution?