How are number of mappers determined for a query with hive on tez?

[#1]

I am looking into a simple select count(*) query based by avro. If we use mapreduce, I see around 50 mappers spawned for this. If we use tez, I see 367 mappers being used. Overall query time increased with more mappers from 55sec to 105 secs.

What factors are determining the number of mappers? What is the best way to reduce the number of mappers in this case? Could it be related to table being in avro format?

[#2]

The following parameters control the number of mappers for splittable formats with Tez:

set tez.grouping.min-size=16777216; -- 16 MB min split
set tez.grouping.max-size=1073741824; -- 1 GB max split

MapReduce uses the following:

set mapreduce.input.fileinputformat.split.minsize=16777216; -- 16 MB
set mapreduce.input.fileinputformat.split.minsize=1073741824; -- 1 GB

Increase min and max split size to reduce the number of mappers.

[#3 · Accepted solution]

MRv2 uses CombineInputFormat, while Tez uses grouped splits. I suspect the table has 367 files, which are not being grouped because the entire cluster has >215 slots - can you confirm the total # of files in the table? Also, it is a good idea to rebuild the statistics - "analyze table <tbl> compute statistics;"

[#4]

I looked at avro and it has 23 files. Grouping min and max are default so its 16MB and 1GB. There were 56 blocks on 24 files and a total size of 300MB. It seemed to have 16MB blocks in grouping since the queue is empty. However, using it ran longer with smaller maps than when it ran with 50 mappers.

[#5]

We have created this write-up some time ago, might be useful: https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works

[#6]

Easiest way to change number of mappers to desired number is:

set tez.grouping.split-count = YOUR-NUMBER-OF-TASKS;

As pointed by Andrew Grande, documented here: https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works

[#7]

Hi,

I have set the tez.grouping.split-count = <Number of mapper count> value but still hive not running that number of mapper while executing query.

Is their any other property also i need to set with tez.grouping.split-count property.