Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Yarn queue Mapping

avatar
Explorer

I have two users,user 1 and user 2 mapped to queue1 and queue 2 respectively in yarn queue Mapping property.

mapreduce.queue.name is set to queue1.

After submitting mapreduce job from user 2, job is getting submitted to queue1 (because of mapreduce queue name).However , I am not expecting this behaviour. My expectation is that user 2 mapreduce job should be submitted to queue 2 and (user 1 to queue 1)directly as per yarn queue Mapping.

I don't want to set queue name , while submitting the job.

Any suggestions on this? How could I achieve desired functionality ?

1 ACCEPTED SOLUTION

avatar
Contributor

@Sushant

If you have mapreduce.job.queuename set to 'AAA', and Queue Mappings Override set to false, when MR job is launched, the queue for the job gets set as AAA. If you launch job, and specify queue as 'batch', it should run in batch queue.

If you set Queue Mappings Override to true, and have yarn.scheduler.capacity.queue-mappings=u:XYZ:batch, and you launch the job as XYZ, without specifying queue, the queue mappings will act as an override and run the job in the batch queue.

Is this the behaviour you are seeing??

View solution in original post

8 REPLIES 8

avatar
Contributor

@Sushant

If you have mapreduce.job.queuename set to 'AAA', and Queue Mappings Override set to false, when MR job is launched, the queue for the job gets set as AAA. If you launch job, and specify queue as 'batch', it should run in batch queue.

If you set Queue Mappings Override to true, and have yarn.scheduler.capacity.queue-mappings=u:XYZ:batch, and you launch the job as XYZ, without specifying queue, the queue mappings will act as an override and run the job in the batch queue.

Is this the behaviour you are seeing??

avatar
Explorer

Current configuration:

Yarn site (Scheduler conf) : yarn.scheduler.capacity.queue-mappings= u:user1: queue1, u: user2: queue 2

Yarn.queue.overriding.enabled: false

Mapred-site.Xml : Mapreduce.queue.name : queue1.

User 2 submitting the mapreduce job is going to queue 1 , but ideally it should go to queue 2( as per mapping). If I set yarn.override property to true, will user2 mapreduce job get submitted to queue 2 directly ? Please suggest/confirm.

avatar
Contributor

Yes. Please try setting yarn queue-mappings-override property to true

avatar
Contributor

@Sushant: Can you please accept the answer if it resolved the issue?

avatar
Rising Star

@Vani This solution works but the side-effect now is that users are allowed to override to which queue their jobs are assigned. Do you agree? Do you in that case know any way around this?

avatar
Rising Star

@Thomas Larsson I agree if I set yarn queue-mappings-override property to true. Then I cannot override queue if I want to for some job. Any solution for that?

avatar
Expert Contributor

You need to set yarn.scheduler.capacity.queue-mappings-override.enable to true, if you want to override the setting from mapred-site.xml (queue 1) with your default mapping (queue 2).

avatar
Cloudera Employee

if yarn.scheduler.capacity.queue-mappings-override.enable is set to false then user's have more control on which queue they want to submit the application either by setting the queue name or from default mapping, in this case user specified queue name takes the precedence , but if yarn.scheduler.capacity.queue-mappings-override.enable set to true then admin's have more control and admin can override user's explicit settings.