Reply
New Contributor
Posts: 3
Registered: ‎10-30-2018
Accepted Solution

Dynamic Resource Pool Configuration

Hi Cloudera Community, 

 

Why Cloudera don't recommended the creation of root.[username] pool on the Resource Pools configuration ? I can't find docummentation realted with this.

 

Best Regards,

 

placement rule.png

 

Cloudera Employee
Posts: 71
Registered: ‎06-10-2015

Re: Dynamic Resource Pool Configuration

Hi Esteban,

 

Our recommendation is to place [username] pools under a parent pool, such as root.users.[username].  This allows you to control overall usage of [username] queues relative to other pools under the root.  This way, if you have 3 root level pools:

 

root.production

root.adhoc

root.users

 

...you can define appropriate weights at amonst these 3 pools.

 

If you configured root.[username], each user pool will be added with a default share of 1.  So, for example, if your initial configuration was:

 

Weight  Pool

10         root.production

10         root.adhoc

...          root.[username] placement rule

 

In the beginning, root.production and root.adhoc will each have 50% of cluster resources.  When the first user runs a job, their subpool is created with a default weight of 1.

 

Weight  Pool

10         root.production

10         root.adhoc

1           root.user1

 

Now imagine that 50 users run a job and 50 new pools are created at the root level with weight of 1.  All of the sudden you have weighted resources heavily to the user pools:

 

Weight  Pool

10         root.production

10         root.adhoc

50          root.[username] generated pools (with 50 users)

 

So in summary, you may prefer to do something that would limit all users, no matter how many, to a fixed ratio of cluster resources:

 

Weight  Pool

10         root.production

10         root.adhoc

10         root.users

             -> root.users.[username] placement rule

 

Thanks,

Nick

Announcements