Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Dynamic Resource Pool Configuration

avatar
New Contributor

Hi Cloudera Community, 

 

Why Cloudera don't recommended the creation of root.[username] pool on the Resource Pools configuration ? I can't find docummentation realted with this.

 

Best Regards,

 

placement rule.png

 

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Hi Esteban,

 

Our recommendation is to place [username] pools under a parent pool, such as root.users.[username].  This allows you to control overall usage of [username] queues relative to other pools under the root.  This way, if you have 3 root level pools:

 

root.production

root.adhoc

root.users

 

...you can define appropriate weights at amonst these 3 pools.

 

If you configured root.[username], each user pool will be added with a default share of 1.  So, for example, if your initial configuration was:

 

Weight  Pool

10         root.production

10         root.adhoc

...          root.[username] placement rule

 

In the beginning, root.production and root.adhoc will each have 50% of cluster resources.  When the first user runs a job, their subpool is created with a default weight of 1.

 

Weight  Pool

10         root.production

10         root.adhoc

1           root.user1

 

Now imagine that 50 users run a job and 50 new pools are created at the root level with weight of 1.  All of the sudden you have weighted resources heavily to the user pools:

 

Weight  Pool

10         root.production

10         root.adhoc

50          root.[username] generated pools (with 50 users)

 

So in summary, you may prefer to do something that would limit all users, no matter how many, to a fixed ratio of cluster resources:

 

Weight  Pool

10         root.production

10         root.adhoc

10         root.users

             -> root.users.[username] placement rule

 

Thanks,

Nick

View solution in original post

1 REPLY 1

avatar
Expert Contributor

Hi Esteban,

 

Our recommendation is to place [username] pools under a parent pool, such as root.users.[username].  This allows you to control overall usage of [username] queues relative to other pools under the root.  This way, if you have 3 root level pools:

 

root.production

root.adhoc

root.users

 

...you can define appropriate weights at amonst these 3 pools.

 

If you configured root.[username], each user pool will be added with a default share of 1.  So, for example, if your initial configuration was:

 

Weight  Pool

10         root.production

10         root.adhoc

...          root.[username] placement rule

 

In the beginning, root.production and root.adhoc will each have 50% of cluster resources.  When the first user runs a job, their subpool is created with a default weight of 1.

 

Weight  Pool

10         root.production

10         root.adhoc

1           root.user1

 

Now imagine that 50 users run a job and 50 new pools are created at the root level with weight of 1.  All of the sudden you have weighted resources heavily to the user pools:

 

Weight  Pool

10         root.production

10         root.adhoc

50          root.[username] generated pools (with 50 users)

 

So in summary, you may prefer to do something that would limit all users, no matter how many, to a fixed ratio of cluster resources:

 

Weight  Pool

10         root.production

10         root.adhoc

10         root.users

             -> root.users.[username] placement rule

 

Thanks,

Nick