Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How are number of mappers determined for a query with hive on tez?

Solved Go to solution
Highlighted

How are number of mappers determined for a query with hive on tez?

Guru

I am looking into a simple select count(*) query based by avro. If we use mapreduce, I see around 50 mappers spawned for this. If we use tez, I see 367 mappers being used. Overall query time increased with more mappers from 55sec to 105 secs.

What factors are determining the number of mappers? What is the best way to reduce the number of mappers in this case? Could it be related to table being in avro format?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How are number of mappers determined for a query with hive on tez?

Rising Star

MRv2 uses CombineInputFormat, while Tez uses grouped splits. I suspect the table has 367 files, which are not being grouped because the entire cluster has >215 slots - can you confirm the total # of files in the table? Also, it is a good idea to rebuild the statistics - "analyze table <tbl> compute statistics;"

6 REPLIES 6

Re: How are number of mappers determined for a query with hive on tez?

Contributor

The following parameters control the number of mappers for splittable formats with Tez:

set tez.grouping.min-size=16777216; -- 16 MB min split
set tez.grouping.max-size=1073741824; -- 1 GB max split

MapReduce uses the following:

set mapreduce.input.fileinputformat.split.minsize=16777216; -- 16 MB
set mapreduce.input.fileinputformat.split.minsize=1073741824; -- 1 GB

Increase min and max split size to reduce the number of mappers.

Re: How are number of mappers determined for a query with hive on tez?

Rising Star

MRv2 uses CombineInputFormat, while Tez uses grouped splits. I suspect the table has 367 files, which are not being grouped because the entire cluster has >215 slots - can you confirm the total # of files in the table? Also, it is a good idea to rebuild the statistics - "analyze table <tbl> compute statistics;"

Re: How are number of mappers determined for a query with hive on tez?

Guru

I looked at avro and it has 23 files. Grouping min and max are default so its 16MB and 1GB. There were 56 blocks on 24 files and a total size of 300MB. It seemed to have 16MB blocks in grouping since the queue is empty. However, using it ran longer with smaller maps than when it ran with 50 mappers.

Re: How are number of mappers determined for a query with hive on tez?

We have created this write-up some time ago, might be useful: https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works

Re: How are number of mappers determined for a query with hive on tez?

Easiest way to change number of mappers to desired number is:

set tez.grouping.split-count = YOUR-NUMBER-OF-TASKS;

As pointed by Andrew Grande, documented here: https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works

Re: How are number of mappers determined for a query with hive on tez?

New Contributor

Hi,

I have set the tez.grouping.split-count = <Number of mapper count> value but still hive not running that number of mapper while executing query.

Is their any other property also i need to set with tez.grouping.split-count property.