Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn queue Mapping

Solved Go to solution

Yarn queue Mapping

New Contributor

I have two users,user 1 and user 2 mapped to queue1 and queue 2 respectively in yarn queue Mapping property.

mapreduce.queue.name is set to queue1.

After submitting mapreduce job from user 2, job is getting submitted to queue1 (because of mapreduce queue name).However , I am not expecting this behaviour. My expectation is that user 2 mapreduce job should be submitted to queue 2 and (user 1 to queue 1)directly as per yarn queue Mapping.

I don't want to set queue name , while submitting the job.

Any suggestions on this? How could I achieve desired functionality ?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Yarn queue Mapping

Contributor

@Sushant

If you have mapreduce.job.queuename set to 'AAA', and Queue Mappings Override set to false, when MR job is launched, the queue for the job gets set as AAA. If you launch job, and specify queue as 'batch', it should run in batch queue.

If you set Queue Mappings Override to true, and have yarn.scheduler.capacity.queue-mappings=u:XYZ:batch, and you launch the job as XYZ, without specifying queue, the queue mappings will act as an override and run the job in the batch queue.

Is this the behaviour you are seeing??

8 REPLIES 8

Re: Yarn queue Mapping

Contributor

@Sushant

If you have mapreduce.job.queuename set to 'AAA', and Queue Mappings Override set to false, when MR job is launched, the queue for the job gets set as AAA. If you launch job, and specify queue as 'batch', it should run in batch queue.

If you set Queue Mappings Override to true, and have yarn.scheduler.capacity.queue-mappings=u:XYZ:batch, and you launch the job as XYZ, without specifying queue, the queue mappings will act as an override and run the job in the batch queue.

Is this the behaviour you are seeing??

Re: Yarn queue Mapping

New Contributor

Current configuration:

Yarn site (Scheduler conf) : yarn.scheduler.capacity.queue-mappings= u:user1: queue1, u: user2: queue 2

Yarn.queue.overriding.enabled: false

Mapred-site.Xml : Mapreduce.queue.name : queue1.

User 2 submitting the mapreduce job is going to queue 1 , but ideally it should go to queue 2( as per mapping). If I set yarn.override property to true, will user2 mapreduce job get submitted to queue 2 directly ? Please suggest/confirm.

Re: Yarn queue Mapping

Contributor

Yes. Please try setting yarn queue-mappings-override property to true

Re: Yarn queue Mapping

Contributor

@Sushant: Can you please accept the answer if it resolved the issue?

Re: Yarn queue Mapping

Contributor

@Vani This solution works but the side-effect now is that users are allowed to override to which queue their jobs are assigned. Do you agree? Do you in that case know any way around this?

Re: Yarn queue Mapping

Rising Star

@Thomas Larsson I agree if I set yarn queue-mappings-override property to true. Then I cannot override queue if I want to for some job. Any solution for that?

Re: Yarn queue Mapping

Expert Contributor

You need to set yarn.scheduler.capacity.queue-mappings-override.enable to true, if you want to override the setting from mapred-site.xml (queue 1) with your default mapping (queue 2).

Re: Yarn queue Mapping

Cloudera Employee

if yarn.scheduler.capacity.queue-mappings-override.enable is set to false then user's have more control on which queue they want to submit the application either by setting the queue name or from default mapping, in this case user specified queue name takes the precedence , but if yarn.scheduler.capacity.queue-mappings-override.enable set to true then admin's have more control and admin can override user's explicit settings.

Don't have an account?
Coming from Hortonworks? Activate your account here