Skip to main content
Scout
Gold ⭐️
January 26, 2023
Question

CDP 7 placement rules dont work

Hi, trying to set placement rules as described in https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/yarn-allocate-resources/topics/yarn-placement-rule-policies.html, but tasks still in default queue.

Also tried to enable tag-based authorization with the same result.

What could be wrong?

14 replies

Asok
Gold ⭐️
Gold ⭐️
February 26, 2023

Please share a screenshot of your placement rules

Translate
mimran
Silver ⭐️
Silver ⭐️
February 27, 2023

Hello @Scout 

First, we need to understand we tried to enable tag-based policy only for Hive jobs because those jobs submitted by proxy user hive. After we enable the tab-bases policy the job submitted with hive user instead of proxy user

Instructions

Step 1:

Enable application tag placement and whitelist hive

  1. In Cloudera Manager, select the YARN service.
  2. Click the Configuration tab.
  3. Search for ResourceManager. In the Filters pane, under Scope, select ResourceManager.
  4. In ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml add the following.
  5. Set these two yarn configurations:
  • yarn.resourcemanager.application-tag-based-placement.enable=true;
  • yarn.resourcemanager.application-tag-based-placement.username.whitelist=hive;

 

Step 2:

You need to check, and if not found, add several configuration properties to allow placement of the Hive workload on the Yarn queue manager.

  1. In Cloudera Manager, click Clusters > Hive on Tez> Configuration.
  2. Search for the Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml setting.
  3. In the Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml setting, click +.
  • In Name enter the property hive.server2.tez.initialize.default.sessions and in value enter false.
  • In Name enter the property hive.server2.tez.queue.access.check and in value enter true.
  • In Name enter the property hive.server2.tez.sessions.custom.queue.allowed and in value enter true.

 

Next you need to add placement rules as below:

Add placement rules >> This rule matches = hive_user >> Custom policy = <yarn.queue.name> (where you want to submit the job.

 

Note: Order of policy also played vital role, please order properly to move the job in aligned queue.

Translate
Scout
ScoutAuthor
Gold ⭐️
March 24, 2023

Hello, sorry for a big delay.

That solution didn't worked for me.

I've set all parameters in Yarn and Hive-on-Tez, but test user going to default query anyway.

Maybe I can check logs or something for that placement rules?

placeRule.png

Translate
VidyaSargur
Community Manager
Community Manager
March 2, 2023

@Scout, Has any of the replies helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  

Translate
Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
Scout
ScoutAuthor
Gold ⭐️
March 29, 2023

Nope.

We can set queue for spark with --queue argument and it would work.

But the rest requests are ignoring yarn placement rules.

Translate
Asok
Gold ⭐️
Gold ⭐️
April 1, 2023

Please share a specific use case, ie. Pick a sample user job that is expected to go to a specific queue and share its placement rules screenshot

Translate
Scout
ScoutAuthor
Gold ⭐️
April 5, 2023

Tried to run spark and zeppelin jobs.

application placement rules are:

-submitting user's name = <user>

-place the application into queue named for user / set default queue to / use custom placement rules

- tried to use static and dynamic queues

The results are the same: application runs in default queue

Translate
Salome23
Silver ⭐️⭐️
April 1, 2023

Hi, I am not able to open a case but I work like a agent for cloudera , I was running the stock market and I am not able to withdraw my deposit since the system give me high package and I can’t manage to cover the overdraft again.. what can I do? 

Translate
Salome23
Silver ⭐️⭐️
April 1, 2023

3A97F35E-3164-497D-BA19-AFE217E3A132.jpeg

  someone who can help me with this, I don’t want to lose my money? 

Translate
JeremyBertelli
Gold ⭐️
Gold ⭐️
April 5, 2023

You do not have any value for the user name. "This rule mactches: users named <your_user>".
When you fill the rule you should set the name (or group, or application name) of the user that match your placement rule.

Translate
Scout
ScoutAuthor
Gold ⭐️
April 7, 2023

Username is set, but deteted due to security reasosns

Translate
Scout
ScoutAuthor
Gold ⭐️
April 11, 2023

found in yarn logs records like 

failed to save service-tags to cache file '/var/lib/ranger/yarn/policy-cache/yarn_cm_yarn_tag.json'
java.io.FileNotFoundException: /var/lib/ranger/yarn/policy-cache/yarn_cm_yarn_tag.json (Permission denied)

Could it be the core reason? 

How can i fix it?

Translate
AsimShaikh
Gold ⭐️
Gold ⭐️
April 13, 2023

Hi @Scout 

Can you share acl for 

/var/lib/ranger/yarn/policy-cache/yarn_cm_yarn_tag.json

Translate
Scout
ScoutAuthor
Gold ⭐️
April 17, 2023

ls: cannot access '/var/lib/ranger/yarn/': No such file or directory

Translate