question Re: Yarn queue Mapping in Support Questions

Yarn queue Mapping

hadcloudera — Wed, 31 May 2017 22:32:49 GMT

I have two users,user 1 and user 2 mapped to queue1 and queue 2 respectively in yarn queue Mapping property.

mapreduce.queue.name is set to queue1.

After submitting mapreduce job from user 2, job is getting submitted to queue1 (because of mapreduce queue name).However , I am not expecting this behaviour. My expectation is that user 2 mapreduce job should be submitted to queue 2 and (user 1 to queue 1)directly as per yarn queue Mapping.

I don't want to set queue name , while submitting the job.

Any suggestions on this? How could I achieve desired functionality ?

Re: Yarn queue Mapping

vasubramanian — Wed, 31 May 2017 23:35:03 GMT

@Sushant

If you have mapreduce.job.queuename set to 'AAA', and Queue Mappings Override set to false, when MR job is launched, the queue for the job gets set as AAA. If you launch job, and specify queue as 'batch', it should run in batch queue.

If you set Queue Mappings Override to true, and have yarn.scheduler.capacity.queue-mappings=u:XYZ:batch, and you launch the job as XYZ, without specifying queue, the queue mappings will act as an override and run the job in the batch queue.

Is this the behaviour you are seeing??

Re: Yarn queue Mapping

hadcloudera — Wed, 31 May 2017 23:51:13 GMT

Current configuration:

Yarn site (Scheduler conf) : yarn.scheduler.capacity.queue-mappings= u:user1: queue1, u: user2: queue 2

Yarn.queue.overriding.enabled: false

Mapred-site.Xml : Mapreduce.queue.name : queue1.

User 2 submitting the mapreduce job is going to queue 1 , but ideally it should go to queue 2( as per mapping). If I set yarn.override property to true, will user2 mapreduce job get submitted to queue 2 directly ? Please suggest/confirm.

Re: Yarn queue Mapping

gnovak — Thu, 01 Jun 2017 13:39:22 GMT

You need to set yarn.scheduler.capacity.queue-mappings-override.enable to true, if you want to override the setting from mapred-site.xml (queue 1) with your default mapping (queue 2).

Re: Yarn queue Mapping

vasubramanian — Thu, 01 Jun 2017 13:49:32 GMT

Yes. Please try setting yarn queue-mappings-override property to true

Re: Yarn queue Mapping

vasubramanian — Tue, 06 Jun 2017 22:38:37 GMT

@Sushant: Can you please accept the answer if it resolved the issue?

Re: Yarn queue Mapping

ThomasLarsson — Fri, 09 Jun 2017 18:01:54 GMT

@Vani This solution works but the side-effect now is that users are allowed to override to which queue their jobs are assigned. Do you agree? Do you in that case know any way around this?

Re: Yarn queue Mapping

shashant_panwar — Wed, 13 Sep 2017 20:49:46 GMT

@Thomas Larsson I agree if I set yarn queue-mappings-override property to true. Then I cannot override queue if I want to for some job. Any solution for that?

Re: Yarn queue Mapping

smunigati — Thu, 12 Oct 2017 22:42:45 GMT

if yarn.scheduler.capacity.queue-mappings-override.enable is set to false then user's have more control on which queue they want to submit the application either by setting the queue name or from default mapping, in this case user specified queue name takes the precedence , but if yarn.scheduler.capacity.queue-mappings-override.enable set to true then admin's have more control and admin can override user's explicit settings.