- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
hive set map.reduce.tasks not working
Created ‎06-27-2017 09:54 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i am setting this parameter to use 32 mappers but its still using one
hive> set map.reduce.tasks=32
> explain select sum(amnt) from lanetxn; OK Plan not optimized by CBO. Vertex dependency in root stage Reducer 2 <- Map 1 (SIMPLE_EDGE) Stage-0 Fetch Operator limit:-1 Stage-1 Reducer 2
Created ‎06-28-2017 04:26 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Sami Ahmad try hive> set mapred.map.tasks = 32;
Created ‎06-28-2017 01:29 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
still the same issue , its still using only one map task and 2 reduce tasks.
Created ‎07-01-2017 03:44 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Maximum number of mappers will be bound by the number of splits calculated at split generation time.
These settings impact split calculation:
- mapreduce.input.fileinputformat.split.minsize
- mapreduce.input.fileinputformat.split.maxsize
Splits are grouped at the Tez layer based on these settings:
- tez.grouping.min-size
- tez.grouping.max-size
If you want more mappers you can tune all these settings down. Note this will not guarantee lower latency, especially on small clusters.
