question Re: How are number of mappers determined for a query with hive on tez? in Archives of Support Questions (Read Only)

How are number of mappers determined for a query with hive on tez?

ravi1 — Tue, 06 Oct 2015 00:13:24 GMT

I am looking into a simple select count(*) query based by avro. If we use mapreduce, I see around 50 mappers spawned for this. If we use tez, I see 367 mappers being used. Overall query time increased with more mappers from 55sec to 105 secs.

What factors are determining the number of mappers? What is the best way to reduce the number of mappers in this case? Could it be related to table being in avro format?

Re: How are number of mappers determined for a query with hive on tez?

jpp — Tue, 06 Oct 2015 00:38:22 GMT

The following parameters control the number of mappers for splittable formats with Tez:

set tez.grouping.min-size=16777216; -- 16 MB min split
set tez.grouping.max-size=1073741824; -- 1 GB max split

MapReduce uses the following:

set mapreduce.input.fileinputformat.split.minsize=16777216; -- 16 MB
set mapreduce.input.fileinputformat.split.minsize=1073741824; -- 1 GB

Increase min and max split size to reduce the number of mappers.

Re: How are number of mappers determined for a query with hive on tez?

gopalv — Tue, 06 Oct 2015 01:35:56 GMT

MRv2 uses CombineInputFormat, while Tez uses grouped splits. I suspect the table has 367 files, which are not being grouped because the entire cluster has >215 slots - can you confirm the total # of files in the table? Also, it is a good idea to rebuild the statistics - "analyze table <tbl> compute statistics;"

Re: How are number of mappers determined for a query with hive on tez?

ravi1 — Thu, 08 Oct 2015 05:19:15 GMT

I looked at avro and it has 23 files. Grouping min and max are default so its 16MB and 1GB. There were 56 blocks on 24 files and a total size of 300MB. It seemed to have 16MB blocks in grouping since the queue is empty. However, using it ran longer with smaller maps than when it ran with 50 mappers.

Re: How are number of mappers determined for a query with hive on tez?

andrewg — Thu, 08 Oct 2015 20:04:23 GMT

We have created this write-up some time ago, might be useful: https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works

Re: How are number of mappers determined for a query with hive on tez?

gbraccialli3 — Sun, 15 Nov 2015 09:09:04 GMT

Easiest way to change number of mappers to desired number is:

set tez.grouping.split-count = YOUR-NUMBER-OF-TASKS;

As pointed by Andrew Grande, documented here: https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works

Re: How are number of mappers determined for a query with hive on tez?

mandar2174 — Mon, 02 May 2016 12:39:54 GMT

Hi,

I have set the tez.grouping.split-count = <Number of mapper count> value but still hive not running that number of mapper while executing query.

Is their any other property also i need to set with tez.grouping.split-count property.