Support Questions
Find answers, ask questions, and share your expertise

Resource Pool placements rules not applied

New Contributor

Hi all,

I have Cloudera 5.11.1.

I have created dynamic resource pool and applied some placemente rules, but for some reason it seems that they aren't applied.

 

Example test:

We have these users test:

1. test_pool1_test1

2. test_pool2_test1

3. test_pool2_test2

4. test_nopool_user1

5. hive

 

Here our placement rules:

1 Use the pool root.[username], only if the pool exists.
2 Use the pool root.test_pool1.[username], only if the pool exists.
3 Use the pool root.test_pool2.[username], only if the pool exists.
4 Use the pool root.users.[username] and create the pool if it does not exist.
5 Use the pool Specified at run time and create the pool if it does not exist.
6 Use the pool root.default.

 

Our pools are configured as following:

* root

   + test_pool1   (with weight 5)
       - test_pool1_test1                       <-- pool associate to a specific user name 
   + test_pool2   (with weight 2)
       - test_pool2_test1                       <-- pool associate to a specific user name
       - test_pool2_test2                       <-- pool associate to a specific user name

   + hive (with weight 10)

   + users (with weight 7)

*weight are random values 🙂

 

When we run some jobs, we noticed:

1. all user jobs (test_pool1_test1,test_pool2_test1,test_pool2_test2,test_nopool_user1) are loaded inside the same pool root.test_pool1.[username]

2. In yarn --> Applications all jobs are loaded in root.test_pool1.[username]

3. users are not able to create subpools in test_pool2 is configured 

4. hive jobs fall correctly in root.hive

 

a.I expected to see test_nopool_user1 inside the root.users.test_nopool_user1 instead of test_pool1.test_nopool_user1 as defined in rule n.4

b.I expected that test_pool2_test1  and test_pool2_test2 loaded jobs inside root.test_pool2.test_pool2_test1 and root.test_pool2.test_pool2_test2 instead of root.test_pool1.test_pool2_test1 and root.test_pool1+.test_pool2_test2 (thay also cannot create subpools in root.test_pool1) as defined in rule n.3

c. User hive appliy correctly rule n.1

 

Any ideas? Have I configured something wrong?

Thank you

G.

 

 

 

 

 

 

 

3 REPLIES 3

Explorer

There is always a resource pool named root.default. By default, all YARN applications run in this pool. You create additional pools when your workload includes identifiable groups of applications (such as from a particular application, or a particular group within your organization) that have their own requirements.

New Contributor

Hi Lewis,

thank you for the answer.

My problemis that all users (except hive) fall into root.test_pool1.[username] even if they have their own subpool in root.test_pool2.

The "default" rule is the last rule configured, so I expect that no jobs match this rule.

Is it correct?

 

My rules workflow should work as follow:

for user hive

1 Use the pool root.hive, only if the pool exists.

    Match, apply rule, the job start in root.hive and Skip other rules

for user test_pool1_test1

1. Use the pool root.test_pool1_test1, only if the pool exists.

    no, skip rule

2 Use the pool root.test_pool1.test_pool1_test1, only if the pool exists.

    Match, apply rule, the job start in root.test_pool1.test_pool1_test1 pool  and skip  other rules

for user test_pool2_test1

1 Use the pool root.test_pool2_test1, only if the pool exists.

    no, skip rule
2 Use the pool root.test_pool1.test_pool2_test1, only if the pool exists.

    no, skip rule
3 Use the pool root.test_pool2.test_pool2_test1, only if the pool exists.

    Match, apply rule, the job start in root.test_pool2.test_pool2_test1 and skip  other rules

for user test_pool2_test2

1 Use the pool root.test_pool2_test2, only if the pool exists.

    no, skip rule
2 Use the pool root.test_pool1.test_pool2_test2, only if the pool exists.

    no, skip rule
3 Use the pool root.test_pool2.test_pool2_test2, only if the pool exists.

    Match, apply rule, the job start in root.test_pool2.test_pool2_test2 and skip other rules

for user test_nopool_user1

1 Use the pool root.test_nopool_user1, only if the pool exists.

    no, skip rule
2 Use the pool root.test_pool1.test_nopool_user1, only if the pool exists.

    no, skip rule
3 Use the pool root.test_pool2.test_nopool_user1, only if the pool exists.

    no, skip rule
4 Use the pool root.users.test_nopool_user1 and create the pool if it does not exist.

    Match, apply rule, create the subpool in root.users.test_nopool_user1 and skip other rules

 

This should be the behaviour I expect, but it doesn't apply

 

New Contributor

Moreover I also configured in pools the "Allow these users and groups to submit to this pool" in the Submission Access control section specifying only the user authorized su submit job to these pools.

 

root.test_pool1  --> allow only users "test_pool1_test1"

  + root.test_pool1.test_pool1_test1 --> allow all (specified "Allow anyone to submit to this pool")

root.test_pool2  --> allow only users "test_pool2_test1,test_pool2_test2"

  + root.test_pool2.test_pool2_test1 --> allow all (specified "Allow anyone to submit to this pool")

  + root.test_pool2.test_pool2_test2 --> allow all (specified "Allow anyone to submit to this pool")

root.hive --> allow only users "hive"

users  --> allow all (specified "Allow anyone to submit to this pool")