Yarn queue Mapping

[#1]

I have two users,user 1 and user 2 mapped to queue1 and queue 2 respectively in yarn queue Mapping property.

mapreduce.queue.name is set to queue1.

After submitting mapreduce job from user 2, job is getting submitted to queue1 (because of mapreduce queue name).However , I am not expecting this behaviour. My expectation is that user 2 mapreduce job should be submitted to queue 2 and (user 1 to queue 1)directly as per yarn queue Mapping.

I don't want to set queue name , while submitting the job.

Any suggestions on this? How could I achieve desired functionality ?

[#2 · Accepted solution]

@Sushant

If you have mapreduce.job.queuename set to 'AAA', and Queue Mappings Override set to false, when MR job is launched, the queue for the job gets set as AAA. If you launch job, and specify queue as 'batch', it should run in batch queue.

If you set Queue Mappings Override to true, and have yarn.scheduler.capacity.queue-mappings=u:XYZ:batch, and you launch the job as XYZ, without specifying queue, the queue mappings will act as an override and run the job in the batch queue.

Is this the behaviour you are seeing??

[#3]

Current configuration:

Yarn site (Scheduler conf) : yarn.scheduler.capacity.queue-mappings= u:user1: queue1, u: user2: queue 2

Yarn.queue.overriding.enabled: false

Mapred-site.Xml : Mapreduce.queue.name : queue1.

User 2 submitting the mapreduce job is going to queue 1 , but ideally it should go to queue 2( as per mapping). If I set yarn.override property to true, will user2 mapreduce job get submitted to queue 2 directly ? Please suggest/confirm.

[#4]

Yes. Please try setting yarn queue-mappings-override property to true

[#5]

@Sushant: Can you please accept the answer if it resolved the issue?

[#6]

@Vani This solution works but the side-effect now is that users are allowed to override to which queue their jobs are assigned. Do you agree? Do you in that case know any way around this?

[#7]

@Thomas Larsson I agree if I set yarn queue-mappings-override property to true. Then I cannot override queue if I want to for some job. Any solution for that?

[#8]

You need to set yarn.scheduler.capacity.queue-mappings-override.enable to true, if you want to override the setting from mapred-site.xml (queue 1) with your default mapping (queue 2).

[#9]

if yarn.scheduler.capacity.queue-mappings-override.enable is set to false then user's have more control on which queue they want to submit the application either by setting the queue name or from default mapping, in this case user specified queue name takes the precedence , but if yarn.scheduler.capacity.queue-mappings-override.enable set to true then admin's have more control and admin can override user's explicit settings.