Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Dynamic Resource Pool Configuration

Solved Go to solution
Highlighted

Dynamic Resource Pool Configuration

New Contributor

Hi Cloudera Community, 

 

Why Cloudera don't recommended the creation of root.[username] pool on the Resource Pools configuration ? I can't find docummentation realted with this.

 

Best Regards,

 

placement rule.png

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Dynamic Resource Pool Configuration

Cloudera Employee

Hi Esteban,

 

Our recommendation is to place [username] pools under a parent pool, such as root.users.[username].  This allows you to control overall usage of [username] queues relative to other pools under the root.  This way, if you have 3 root level pools:

 

root.production

root.adhoc

root.users

 

...you can define appropriate weights at amonst these 3 pools.

 

If you configured root.[username], each user pool will be added with a default share of 1.  So, for example, if your initial configuration was:

 

Weight  Pool

10         root.production

10         root.adhoc

...          root.[username] placement rule

 

In the beginning, root.production and root.adhoc will each have 50% of cluster resources.  When the first user runs a job, their subpool is created with a default weight of 1.

 

Weight  Pool

10         root.production

10         root.adhoc

1           root.user1

 

Now imagine that 50 users run a job and 50 new pools are created at the root level with weight of 1.  All of the sudden you have weighted resources heavily to the user pools:

 

Weight  Pool

10         root.production

10         root.adhoc

50          root.[username] generated pools (with 50 users)

 

So in summary, you may prefer to do something that would limit all users, no matter how many, to a fixed ratio of cluster resources:

 

Weight  Pool

10         root.production

10         root.adhoc

10         root.users

             -> root.users.[username] placement rule

 

Thanks,

Nick

1 REPLY 1

Re: Dynamic Resource Pool Configuration

Cloudera Employee

Hi Esteban,

 

Our recommendation is to place [username] pools under a parent pool, such as root.users.[username].  This allows you to control overall usage of [username] queues relative to other pools under the root.  This way, if you have 3 root level pools:

 

root.production

root.adhoc

root.users

 

...you can define appropriate weights at amonst these 3 pools.

 

If you configured root.[username], each user pool will be added with a default share of 1.  So, for example, if your initial configuration was:

 

Weight  Pool

10         root.production

10         root.adhoc

...          root.[username] placement rule

 

In the beginning, root.production and root.adhoc will each have 50% of cluster resources.  When the first user runs a job, their subpool is created with a default weight of 1.

 

Weight  Pool

10         root.production

10         root.adhoc

1           root.user1

 

Now imagine that 50 users run a job and 50 new pools are created at the root level with weight of 1.  All of the sudden you have weighted resources heavily to the user pools:

 

Weight  Pool

10         root.production

10         root.adhoc

50          root.[username] generated pools (with 50 users)

 

So in summary, you may prefer to do something that would limit all users, no matter how many, to a fixed ratio of cluster resources:

 

Weight  Pool

10         root.production

10         root.adhoc

10         root.users

             -> root.users.[username] placement rule

 

Thanks,

Nick