Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hive set map.reduce.tasks not working

hive set map.reduce.tasks not working

Master Collaborator

i am setting this parameter to use 32 mappers but its still using one

hive> set map.reduce.tasks=32
    > explain select sum(amnt) from lanetxn;
OK
Plan not optimized by CBO.


Vertex dependency in root stage
Reducer 2 <- Map 1 (SIMPLE_EDGE)


Stage-0
   Fetch Operator
      limit:-1
      Stage-1
         Reducer 2


3 REPLIES 3

Re: hive set map.reduce.tasks not working

Contributor

@Sami Ahmad try hive> set mapred.map.tasks = 32;

Re: hive set map.reduce.tasks not working

Master Collaborator

still the same issue , its still using only one map task and 2 reduce tasks.

Highlighted

Re: hive set map.reduce.tasks not working

New Contributor

Maximum number of mappers will be bound by the number of splits calculated at split generation time.

These settings impact split calculation:

  1. mapreduce.input.fileinputformat.split.minsize
  2. mapreduce.input.fileinputformat.split.maxsize

Splits are grouped at the Tez layer based on these settings:

  1. tez.grouping.min-size
  2. tez.grouping.max-size

If you want more mappers you can tune all these settings down. Note this will not guarantee lower latency, especially on small clusters.