12-19-2013 06:04 AM
We have a couple questions around scheduler configuration options. We’ve had a couple of incidents where a single user’s job blocked other user’s job by consuming all of the mappers. The jobs finished however because those jobs were tied to a web application, the web interface timed-out. We are also concerned that in the future large analytical MR jobs could consume cpu/memory and severely impact search jobs that are tied to a web interface and timeout issues could occur. Currently we are using FIFO scheduler. We want to give priority to the web application jobs and limit other analytic jobs.
Answers to these specific questions would be helpful:
12-20-2013 09:36 AM
That's correct that the FIFO Scheduler does not provide mechanisms to avoid these conflicts. We recommend moving to the Fair Scheduler for multitenancy. While the YARN/MR2 fair scheduler has additional features (hierarchical pools, multi-resource scheduling), the MR1 Fair Scheduler will likely support your use case as well.
All you need to do to start using it is to change a single property in your JobTracker's configuration and add a fair-scheduler.xml allocations file. Here's the documentation on the MR1 Fair Scheduler.
If you have any additional questions on it I'd be happy to try to answer them.