question Re: Yarn Scheduling Management, Capacity Scheduler in Archives of Support Questions (Read Only)

Yarn Scheduling Management, Capacity Scheduler

ishvari — Mon, 17 Apr 2017 21:17:04 GMT

In Capacity Scheduler, How Resource Manage decide which Job runs on which Queue?

Re: Yarn Scheduling Management, Capacity Scheduler

jsensharma — Mon, 17 Apr 2017 23:05:46 GMT

The following tutorial will help in understanding the same with a real time example with 3 queues ("Support, Marketing, Engineering" Queues): https://hortonworks.com/hadoop-tutorial/configuring-yarn-capacity-scheduler-ambari/#configure-scheduler

Additionally please refer to: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_yarn_resource_mgt/content/setting_up_queues.html

Re: Yarn Scheduling Management, Capacity Scheduler

gnovak — Tue, 18 Apr 2017 15:16:58 GMT

@Ishvari Dhimmar

The direct answer to your question is that it's not the Resource Manager that decides in which queue a job should run. This is specified explicitly by the user. If you don't specify the queue explicitly, then by default the job will be submitted to the "default" queue (if a queue named "default" doesn't exist in your config, an error will be thrown).

How you can specify the queue depends on the application, e.g. for MapReduce jobs it's defined by "-Dmapred.job.queue.name".

Re: Yarn Scheduling Management, Capacity Scheduler

ishvari — Tue, 18 Apr 2017 20:20:06 GMT

Thanks for Reply. I was thinking like this, but confused. Now its clear. Thanks again.