question Re: Is there any UI for fair scheduler of yarn? in Archives of Support Questions (Read Only)

Is there any UI for fair scheduler of yarn?

younggeun_park — Tue, 12 Jan 2016 18:02:52 GMT

Hello,

In hdp-2.3.2.0-2950, there are UIs for capacity scheduler in Ambari:

Yarn > Configs > Scheduler

Yarn Queue Manager menu on the Top

Is there any UI for fair scheduler?

If not, how can I set the fair scheduler?

Both value of yarn.resourcemanager.scheduler.class and fair-scheduler.xml are enough?

Regards,

park

Re: Is there any UI for fair scheduler of yarn?

jstraub — Tue, 12 Jan 2016 19:37:07 GMT

I am not aware of any Fair Scheduler specific UI.

To enable the Fair Scheduler change the scheduler class in yarn-site

<property>
  <name>yarn.resourcemanager.scheduler.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>

Make sure yarn.scheduler.fair.allocation.file is pointing to your fair-scheduler.xml

Check out https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

Re: Is there any UI for fair scheduler of yarn?

maxim_al_pantel — Fri, 20 May 2016 15:38:44 GMT

There is only one setting in UI where you can change scheduler class. Go to Ambari->YARN->Configs->Advanced->Scheduler. In case of the FairScheduler, here it is: yarn.resourcemanager.scheduler.class. it should be set, as @Jonas Straub mentioned, to 'org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler'. But if you want to define some queues, after setting/saving new scheduler type, go to server where the YARN resource manager is running (it could be pull off from Ambari YARN page) and find the folder that contains file 'yarn-site.xml', put there file 'fair-scheduler.xml' with content:

<?xml version="1.0"?>
<allocations>
    <queue name="default">
        <weight>30</weight>
        <schedulingPolicy>fair</schedulingPolicy>
    </queue>
    <queue name="system_queue">
        <weight>70</weight>
        <schedulingPolicy>fifo</schedulingPolicy>
    </queue>
    <queuePlacementPolicy>
        <rule name="specified" create="false" />
        <rule name="default" queue="default" />
    </queuePlacementPolicy>
</allocations>

here 30:70 ration is treated as fair :). Adjust it according to your needs and don't forget to restart YARN in Ambari.

To verify your changes, go to Ambari->YARN->Quick Links->ResourceManager UI and navigate to Scheduler on the left side. There you should see your scheduler type and queue configuration.

If you still have any questions, don't hesitate to ask.

Re: Is there any UI for fair scheduler of yarn?

2rajukv — Fri, 21 Oct 2016 04:43:21 GMT

@Maxim Panteleev do we have option to set default mappings to these queues similar to capacity scheduler queue mappings.

Re: Is there any UI for fair scheduler of yarn?

ssk26 — Fri, 13 Dec 2019 06:31:56 GMT

Hi,

I have tried the same steps in Ambari. I am using HDP 3.1.0.0-78. Here is what I have done:

Step 1: Went to 'Yarn -> Configs -> Advanced -> Scheduler' and set the properties that you have given.

Step 2: Restarted all the affected components in Yarn. However, I am getting an error

java.io.IOException: Failed to initialize FairScheduler

Caused by: org.xml.sax.SAXParseException; lineNumber: 19; columnNumber: 1; XML document structures must start and end within the same entity.

Also, in "Scheduler" component in Yarn, I am only getting "Capacity Scheduler" related parameters. If I add "fair-scheduler.xml" in the same folder as that of "yarn-site.xml" and specify parameters only for "default" queue (Here I am not adding any new queue) only to change its scheduling mode to Fair, how to do it?

Can you please guide me?