Verify yarn scheduler running configuration

[#1]

Hi,

So for most services, you just jump over to <node>:<port>/conf to see the running configuration. Same thing for yarn, except that we're not seeing the scheduler configuration. So, for example, all the queues and acl's don't show up on the resource manager conf page. 

You can just jump over to the scheduler page and expand the different queues and view the options, but we're attempting to specify a different resource calculator for the capacity scheduler, and since it's not behaving as we anticipated, wanted to first verify we were able to change to the other calculator. 

Doni

[#2]

Hi Doni,

That's a great question!  Currently there isn't a straightforward way at the moment to verify the actual runtime value of the Capacity Scheduler's specific properties via Web UI, however it is possible to confirm what the value of the property each specific process was given at the time of YARN services/role startup.  Since the Capacity Scheduler resource calculator is run on the Resource Manager and all the Node Managers, you can spot-check the property given to these roles by doing the following:

For CM-managed clusters:

1) Navigate to CM -> Clusters -> YARN -> Configuration -> Search for yarn.resourcemanager.scheduler.class2) Confirm that the yarn.resourcemanager.scheduler.class has been set to the Capacity Scheduler using the value org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.   Save changes if needed.

3) Navigate to Instances -> (Click on Resource Manager or Node Manager) -> Processes -> Click on capacity-scheduler.xml under Configuration Files.  NOTE:  This property needs to be set on all Resource Manager(s) and  all Node Managers to be effective. 

4)  Search for the property yarn.scheduler.capacity.resource-calculator. If the property is not present here, it will be using the default value of org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator.

5) Confirm that the intended the value that is set, i.e. org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

6) If any changes were made, please Save changes and restart YARN services accordingly and reconfirm the above starting from Step 1.

For CDH-only (no CM) clusters:

1) Confirm that /etc/hadoop/conf/yarn-site.xml for all Resource Manager(s) and Node Managers  contain the entry:

<property>

   <name>yarn.resourcemanager.scheduler.class</name>

   <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>

</property>

2)  Verify that the  yarn.scheduler.capacity.resource-calculator property sets the intended value (for example:  org.apache.hadoop.yarn.util.resource.DominantResourceCalculator) in /etc/hadoop/conf/capacity-scheduler.xml on each of the Resource Manager(s) and all Node Managers.

Hope this helps!

-- Anthony