Yarn Scheduling Management, Capacity Scheduler

[#1]

In Capacity Scheduler, How Resource Manage decide which Job runs on which Queue?

[#2]

@Ishvari Dhimmar

The following tutorial will help in understanding the same with a real time example with 3 queues ("Support, Marketing, Engineering" Queues): https://hortonworks.com/hadoop-tutorial/configuring-yarn-capacity-scheduler-ambari/#configure-schedu...

.

Additionally please refer to: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_yarn_resource_mgt/content/setting_up_que...

.

[#3 · Accepted solution]

@Ishvari Dhimmar

The direct answer to your question is that it's not the Resource Manager that decides in which queue a job should run. This is specified explicitly by the user. If you don't specify the queue explicitly, then by default the job will be submitted to the "default" queue (if a queue named "default" doesn't exist in your config, an error will be thrown).

How you can specify the queue depends on the application, e.g. for MapReduce jobs it's defined by "-Dmapred.job.queue.name".

[#4]

Thanks for Reply. I was thinking like this, but confused. Now its clear. Thanks again.