Reply
Expert Contributor
Posts: 87
Registered: ‎09-17-2014

Maping user group to certain YARN resource pool

Hi dear experts!

 

I'm looking for some way to map cettain user group to YARN resource pool.

for example, i have user Bob, who belongs to group QA and user Alice who Belongs to group Dev

in YARN i have pools like: root.low, root.medium, root.high.

 

I want to config placement rules at certain way that all users who belong to QA group (Bob) would be mapped to root.low pools, and everyone who belongs to Dev (Alice) would map to root.high.

 

does anyone know is it possible or not?

 

thanks!

Posts: 1,748
Kudos: 364
Solutions: 277
Registered: ‎07-31-2013

Re: Maping user group to certain YARN resource pool

See http://blog.cloudera.com/blog/2016/06/untangling-apache-hadoop-yarn-part-4-fair-scheduler-queue-basi...
for help with configuring placement policies in Fair Scheduler via
primary groups/secondary groups.
Expert Contributor
Posts: 87
Registered: ‎09-17-2014

Re: Maping user group to certain YARN resource pool

Thank you so much for sharing this!

but i didn't fing how to map name of Primary/Secondary group to the different Name of the pool.

in other words i want to put user, who belongs to Marketing group into root.low pool, rather than root.marketing

 

thanks!

Posts: 1,748
Kudos: 364
Solutions: 277
Registered: ‎07-31-2013

Re: Maping user group to certain YARN resource pool

I see, sorry for glancing over that aspect of the question. I'm afraid
mapping to a different name is currently not possible in an automatic
manner on the server side.

Would using the group names directly not be possible?
Expert Contributor
Posts: 87
Registered: ‎09-17-2014

Re: Maping user group to certain YARN resource pool

well, I'd like to create only 3 pools with different priorities and put diffrent groups, which could be many (like it could be up to 50 different groups), in different pools (which is only 3)...

i was thinking for some workaround but nothing come to my mind

Posts: 1,748
Kudos: 364
Solutions: 277
Registered: ‎07-31-2013

Re: Maping user group to certain YARN resource pool

You can achieve that with the secondary groups method, with the option that
disables creation of new queue names if they do not yet exist.
Expert Contributor
Posts: 87
Registered: ‎09-17-2014

Re: Maping user group to certain YARN resource pool

thanks for idea.

so if Bob will belong to marketing group and to low group, in case of existing only low pool, it will be mapped to this pool, right?

anohter question, what if i'll have pools:

root.low and root.marketing, which one will Pick up Bob, given that he belongs to both (secondary) groups?

 

Posts: 1,748
Kudos: 364
Solutions: 277
Registered: ‎07-31-2013

Re: Maping user group to certain YARN resource pool

> so if Bob will belong to marketing group and to low group, in case of existing only low pool, it will be mapped to this pool, right?

Yes, that is correct.

> root.low and root.marketing, which one will Pick up Bob, given that he belongs to both (secondary) groups?

This is a good question. Currently the queue selection will be based
on the first group name that matches, in the order of groups returned
by the underlying OS's groups provider. You could say the order is
undefined, but likely consistent.
Highlighted
Expert Contributor
Posts: 87
Registered: ‎09-17-2014

Re: Maping user group to certain YARN resource pool

so, as i can see here we may have solution as disallow undeclared pools and don't create pools which matches with undesirible groops, right?

 

thank you!

Announcements