Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to assigned capacity scheduler queue based on AD group.

Solved Go to solution

How to assigned capacity scheduler queue based on AD group.

Guru

HI,

I have a requirement where we have to assign our CS queues based on ad groups.

For example our ad users are using cluster and running jobs under defined queue but I want that is there any way to configure their AD group with queue so that each member of that queue will go only to a specific queue.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to assigned capacity scheduler queue based on AD group.

@Saurabh Kumar[root@phdns02 scripts]# id neeraj

uid=29800018(neeraj) gid=29800018(neeraj) groups=29800018(neeraj),29800017(hdpadmin)

[root@phdns02 scripts]#

See this

17 REPLIES 17

Re: How to assigned capacity scheduler queue based on AD group.

@Saurabh Kumar

For this you have to make sure that Ambari (If you are using CS view) is in synch with AD. You have to import users and groups that you want to map with queues.

See this demo on queue mapping.

As long as user id matches with CS setup, queues will be in effect.

Re: How to assigned capacity scheduler queue based on AD group.

Guru

Hello @Neeraj Sabharwal: Thanks for the above explanation. I have configured CS view which is working fine for local unix group and users. But when I configured for Ldap or AD group it does not work and fail with below error.

org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1455533826426_0018 to YARN : Failed to submit application application_1455533826426_0018 submitted by user saurkuma reason: No groups found for user saurkuma

Re: How to assigned capacity scheduler queue based on AD group.

@Saurabh Kumar See this "submitted by user saurkuma reason: No groups found for user saurkuma"

Start with user level mapping and then see if your user is part of the group or not.

Re: How to assigned capacity scheduler queue based on AD group.

Guru

@Neeraj Sabharwal: Yes I am a part of the AD group(adhdpadm) and when I configure u:saurkuma:default then it is working fine but when I do g:adhdpadm:default then it is failing with above error.

2112-screen-shot-2016-02-15-at-75702-pm.png

Re: How to assigned capacity scheduler queue based on AD group.

@Saurabh Kumar https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_yarn_resource_mgt/content/default_queue_...

My only concern is if user is part of that group or not.

<property>
  <name>yarn.scheduler.capacity.queue-mappings</name>
  <value>u:maria:engineering,g:webadmins:weblog</value>
</property>

Re: How to assigned capacity scheduler queue based on AD group.

Guru

Hi @Neeraj Sabharwal: when configured ad group mapping then I don't defined any user mapping.And yes user saurkuma is a part of adhdpadm group.

My first point is does it supports ldap groups(Active directory) or not ?

I checked with local unix groups and found them working.

Re: How to assigned capacity scheduler queue based on AD group.

@Saurabh Kumar

Let me run a test And get back to you .

Re: How to assigned capacity scheduler queue based on AD group.

@Saurabh Kumar[root@phdns02 scripts]# id neeraj

uid=29800018(neeraj) gid=29800018(neeraj) groups=29800018(neeraj),29800017(hdpadmin)

[root@phdns02 scripts]#

See this

Re: How to assigned capacity scheduler queue based on AD group.

@Saurabh Kumar You can see that I have mapped hdpadmin group to queue hadoopadmin

user neeraj is part of the group and when I run yarn job, it executes against hdpadmin queue

[root@phdns02 ~]# cat /etc/group | grep hdpadmin

[root@phdns02 ~]# su - neeraj

su: warning: cannot change directory to /home/neeraj: No such file or directory

-sh-4.1$ id

exituid=29800018(neeraj) gid=29800018(neeraj) groups=29800018(neeraj),29800017(hdpadmin) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

-sh-4.1$ exit

logout

[root@phdns02 ~]# id neeraj

uid=29800018(neeraj) gid=29800018(neeraj) groups=29800018(neeraj),29800017(hdpadmin)

[root@phdns02 ~]#

Don't have an account?
Coming from Hortonworks? Activate your account here