Support Questions
Find answers, ask questions, and share your expertise

hive set map.reduce.tasks not working

Master Collaborator

i am setting this parameter to use 32 mappers but its still using one

hive> set map.reduce.tasks=32
    > explain select sum(amnt) from lanetxn;
OK
Plan not optimized by CBO.


Vertex dependency in root stage
Reducer 2 <- Map 1 (SIMPLE_EDGE)


Stage-0
   Fetch Operator
      limit:-1
      Stage-1
         Reducer 2


3 REPLIES 3

Re: hive set map.reduce.tasks not working

Contributor

@Sami Ahmad try hive> set mapred.map.tasks = 32;

Re: hive set map.reduce.tasks not working

Master Collaborator

still the same issue , its still using only one map task and 2 reduce tasks.

Re: hive set map.reduce.tasks not working

Explorer

Maximum number of mappers will be bound by the number of splits calculated at split generation time.

These settings impact split calculation:

  1. mapreduce.input.fileinputformat.split.minsize
  2. mapreduce.input.fileinputformat.split.maxsize

Splits are grouped at the Tez layer based on these settings:

  1. tez.grouping.min-size
  2. tez.grouping.max-size

If you want more mappers you can tune all these settings down. Note this will not guarantee lower latency, especially on small clusters.