Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Maping user group to certain YARN resource pool

Maping user group to certain YARN resource pool

Rising Star

Hi dear experts!

 

I'm looking for some way to map cettain user group to YARN resource pool.

for example, i have user Bob, who belongs to group QA and user Alice who Belongs to group Dev

in YARN i have pools like: root.low, root.medium, root.high.

 

I want to config placement rules at certain way that all users who belong to QA group (Bob) would be mapped to root.low pools, and everyone who belongs to Dev (Alice) would map to root.high.

 

does anyone know is it possible or not?

 

thanks!

8 REPLIES 8
Highlighted

Re: Maping user group to certain YARN resource pool

Master Guru
See http://blog.cloudera.com/blog/2016/06/untangling-apache-hadoop-yarn-part-4-fair-scheduler-queue-basi...
for help with configuring placement policies in Fair Scheduler via
primary groups/secondary groups.

Re: Maping user group to certain YARN resource pool

Rising Star

Thank you so much for sharing this!

but i didn't fing how to map name of Primary/Secondary group to the different Name of the pool.

in other words i want to put user, who belongs to Marketing group into root.low pool, rather than root.marketing

 

thanks!

Re: Maping user group to certain YARN resource pool

Master Guru
I see, sorry for glancing over that aspect of the question. I'm afraid
mapping to a different name is currently not possible in an automatic
manner on the server side.

Would using the group names directly not be possible?

Re: Maping user group to certain YARN resource pool

Rising Star

well, I'd like to create only 3 pools with different priorities and put diffrent groups, which could be many (like it could be up to 50 different groups), in different pools (which is only 3)...

i was thinking for some workaround but nothing come to my mind

Re: Maping user group to certain YARN resource pool

Master Guru
You can achieve that with the secondary groups method, with the option that
disables creation of new queue names if they do not yet exist.

Re: Maping user group to certain YARN resource pool

Rising Star

thanks for idea.

so if Bob will belong to marketing group and to low group, in case of existing only low pool, it will be mapped to this pool, right?

anohter question, what if i'll have pools:

root.low and root.marketing, which one will Pick up Bob, given that he belongs to both (secondary) groups?

 

Re: Maping user group to certain YARN resource pool

Master Guru
> so if Bob will belong to marketing group and to low group, in case of existing only low pool, it will be mapped to this pool, right?

Yes, that is correct.

> root.low and root.marketing, which one will Pick up Bob, given that he belongs to both (secondary) groups?

This is a good question. Currently the queue selection will be based
on the first group name that matches, in the order of groups returned
by the underlying OS's groups provider. You could say the order is
undefined, but likely consistent.

Re: Maping user group to certain YARN resource pool

Rising Star

so, as i can see here we may have solution as disallow undeclared pools and don't create pools which matches with undesirible groops, right?

 

thank you!