- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
mapreduce.job.running.map.limit has no effect
- Labels:
-
Apache Hadoop
Created ‎01-10-2019 04:30 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I want to fix a limit for the number of running mapper per mapreduce job, I set the following parameter in my mapred-site.xml file
<name>mapreduce.job.running.map.limit</name>
<value>25000</value>
But it seems that this parameter has no effect. I found a job which takes 44543 mappers and in the configuration file i found the right parameter with the rigth value (25000).
Is there another parameter to activate this limit?
Created ‎01-11-2019 02:47 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for your answer
I suppose this parameter is only for hive request?
Created ‎01-14-2019 12:20 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No of mappers depends on input split size it cannot be manually configured. Input split is a logical representation of data and block is a physical representation of data. An input split indicates the start and end point of data in a block.
There is one mapper per input split. so the number of mappers is not completely dependent on the number blocks. Depending upon the configured size, the number of splits varies. There might be one split per block, one split per two blocks or maybe two splits per block and so on.
HTH
