Dynamic Resource Pool Configuration

EstebanGonzalez — Fri, 16 Sep 2022 14:16:55 GMT

Hi Cloudera Community,

Why Cloudera don't recommended the creation of root.[username] pool on the Resource Pools configuration ? I can't find docummentation realted with this.

Best Regards,

Re: Dynamic Resource Pool Configuration

nwhite — Fri, 05 Apr 2019 15:46:50 GMT

Hi Esteban,

Our recommendation is to place [username] pools under a parent pool, such as root.users.[username]. This allows you to control overall usage of [username] queues relative to other pools under the root. This way, if you have 3 root level pools:

root.production

root.adhoc

root.users

...you can define appropriate weights at amonst these 3 pools.

If you configured root.[username], each user pool will be added with a default share of 1. So, for example, if your initial configuration was:

Weight Pool

10 root.production

10 root.adhoc

... root.[username] placement rule

In the beginning, root.production and root.adhoc will each have 50% of cluster resources. When the first user runs a job, their subpool is created with a default weight of 1.

Weight Pool

10 root.production

10 root.adhoc

1 root.user1

Now imagine that 50 users run a job and 50 new pools are created at the root level with weight of 1. All of the sudden you have weighted resources heavily to the user pools:

Weight Pool

10 root.production

10 root.adhoc

50 root.[username] generated pools (with 50 users)

So in summary, you may prefer to do something that would limit all users, no matter how many, to a fixed ratio of cluster resources:

Weight Pool

10 root.production

10 root.adhoc

10 root.users

-> root.users.[username] placement rule

Thanks,

Nick

question Re: Dynamic Resource Pool Configuration in Support Questions

Dynamic Resource Pool Configuration

Re: Dynamic Resource Pool Configuration