06-19-2015 11:36 PM
In MRv1 we had the below two configurable parameters to set the number of Map and reduce slots per Node.
Also it was advisable to have number of Map slots little higher than the number of Reduce slots. Ideal number of reducers for a Map Reduce job would be equal to or greater than number of reduce slots available in the cluster.
Please correct if my above understanding is not correct wrt MRv1...
In MRv2 we dont have the concept of slots anymore, instead containers provide the required memory and CPU for Map/Reduce taks execution.
Here comes my question, How to decide on number of reducers for any Map Reduce job in MRv2 ?
06-22-2015 11:31 AM
Below are the details,
12 cores per Node
96 GB memory per Node
12 1TB drives
Also please help me understand how you come up with the numbers?
06-30-2015 07:13 AM
This link should answer your questions.
let me know if you are looking for something else.