question Re: How to assigned capacity scheduler queue based on AD group. in Archives of Support Questions (Read Only)

How to assigned capacity scheduler queue based on AD group.

SK1 — Fri, 12 Feb 2016 21:50:17 GMT

HI,

I have a requirement where we have to assign our CS queues based on ad groups.

For example our ad users are using cluster and running jobs under defined queue but I want that is there any way to configure their AD group with queue so that each member of that queue will go only to a specific queue.

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Fri, 12 Feb 2016 21:55:46 GMT

@Saurabh Kumar

For this you have to make sure that Ambari (If you are using CS view) is in synch with AD. You have to import users and groups that you want to map with queues.

See this demo on queue mapping.

As long as user id matches with CS setup, queues will be in effect.

Re: How to assigned capacity scheduler queue based on AD group.

SK1 — Mon, 15 Feb 2016 21:59:22 GMT

Hello @Neeraj Sabharwal: Thanks for the above explanation. I have configured CS view which is working fine for local unix group and users. But when I configured for Ldap or AD group it does not work and fail with below error.

org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1455533826426_0018 to YARN : Failed to submit application application_1455533826426_0018 submitted by user saurkuma reason: No groups found for user saurkuma

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Mon, 15 Feb 2016 22:02:03 GMT

@Saurabh Kumar See this "submitted by user saurkuma reason: No groups found for user saurkuma"

Start with user level mapping and then see if your user is part of the group or not.

Re: How to assigned capacity scheduler queue based on AD group.

SK1 — Mon, 19 Aug 2019 08:27:28 GMT

@Neeraj Sabharwal: Yes I am a part of the AD group(adhdpadm) and when I configure u:saurkuma:default then it is working fine but when I do g:adhdpadm:default then it is failing with above error.

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Mon, 15 Feb 2016 22:31:06 GMT

@Saurabh Kumar https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_yarn_resource_mgt/content/default_queue_mapping_based_on_user_or_group.html

My only concern is if user is part of that group or not.

<property>
  <name>yarn.scheduler.capacity.queue-mappings</name>
  <value>u:maria:engineering,g:webadmins:weblog</value>
</property>

Re: How to assigned capacity scheduler queue based on AD group.

SK1 — Tue, 16 Feb 2016 00:40:39 GMT

Hi @Neeraj Sabharwal: when configured ad group mapping then I don't defined any user mapping.And yes user saurkuma is a part of adhdpadm group.

My first point is does it supports ldap groups(Active directory) or not ?

I checked with local unix groups and found them working.

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Tue, 16 Feb 2016 00:58:47 GMT

@Saurabh Kumar

Let me run a test And get back to you .

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Tue, 16 Feb 2016 04:14:21 GMT

@Saurabh Kumar[root@phdns02 scripts]# id neeraj

uid=29800018(neeraj) gid=29800018(neeraj) groups=29800018(neeraj),29800017(hdpadmin)

[root@phdns02 scripts]#

See this

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Tue, 16 Feb 2016 04:15:21 GMT

@Saurabh Kumar You can see that I have mapped hdpadmin group to queue hadoopadmin

user neeraj is part of the group and when I run yarn job, it executes against hdpadmin queue

[root@phdns02 ~]# cat /etc/group | grep hdpadmin

[root@phdns02 ~]# su - neeraj

su: warning: cannot change directory to /home/neeraj: No such file or directory

-sh-4.1$ id

exituid=29800018(neeraj) gid=29800018(neeraj) groups=29800018(neeraj),29800017(hdpadmin) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

-sh-4.1$ exit

logout

[root@phdns02 ~]# id neeraj

uid=29800018(neeraj) gid=29800018(neeraj) groups=29800018(neeraj),29800017(hdpadmin)

[root@phdns02 ~]#

Re: How to assigned capacity scheduler queue based on AD group.

SK1 — Wed, 17 Feb 2016 15:22:30 GMT

@Neeraj Sabharwal: Thanks a lot for your testing.

I see you have tested it for a unix user(neeraj) who is part of a unix group(hdpadmin).Which is working fine for me.

But my requirement is we have some users where they don't connect to server,they directly use some tools (like Aqua Data Studio or SQL client or Teradata client) and we validate them by login to our cluster by their LDAP(active directory) with jdbc string or though beeline.

And when they submit their job then they have to set property mapred.job.queue.name and run their jobs.

So my point is can we configure CS view for ldap or AD groups as well ?

I tried it for groups but getting below error. But when I tried for user specific then it is working(as expected)

g:adhdpadm:batch

ERROR : Failed to execute tez graph.

org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1455533826426_0025 to YARN : Failed to submit application application_1455533826426_0025 submitted by user saurkuma reason: No groups found for user saurkuma

u:saurkuma:batch: working

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Wed, 17 Feb 2016 18:05:57 GMT

@Saurabh Kumar

You are incorrect. User neeraj and group hdpadmin is not in UNIX. It's in AD

Look at my comments carefully 🙂

Output of /etc/group and /etc/passwd

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Wed, 17 Feb 2016 18:41:39 GMT

@Saurabh Kumar Give me output of id saurkuma

Re: How to assigned capacity scheduler queue based on AD group.

SK1 — Wed, 17 Feb 2016 20:05:22 GMT

@Neeraj Sabharwal: It might be that same user neeraj is also part of Unix and AD. So thats why it is working. But in my case we have do not have same user in unix group.

please find the below output.

[s0998dnz@*********001 ~]$ id saurkuma

id: saurkuma: No such user

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Wed, 17 Feb 2016 20:07:55 GMT

@Saurabh Kumar

No , neeraj and group is in AD

You can see the output

Also, you need to look into syncing up your server with ad

Use nslcd or sssd

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Mon, 19 Aug 2019 08:27:20 GMT

@Saurabh Kumar following up on this.

user neeraj and group hdpadmin is in LDAP

they dont exists in /etc/passwd and /etc/group

Re: How to assigned capacity scheduler queue based on AD group.

SK1 — Mon, 22 Feb 2016 20:32:40 GMT

@Neeraj Sabharwal: Today I noticed that it is not working as I don't have sssd running and it is not configured also. And I feel it is mandatory for group mapping with hdfs.

I tried to configure sssd with ldap but did not get success, so I need your hep to configure sssd, do you have any doc or instruction to do that ?

Re: How to assigned capacity scheduler queue based on AD group.

nsabharwal — Tue, 15 Mar 2016 02:55:54 GMT

@Saurabh Kumar That's exactly my point was 🙂

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/about-sssd.conf.html

I use this https://github.com/hortonworks-gallery/ambari-nslcd-service

So..As you can see in the demo, we can map AD groups.

https://community.hortonworks.com/articles/17135/yarn-queues-and-ad-group-mapping.html