Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Capacity Scheduler QueueMetrics

Highlighted

Capacity Scheduler QueueMetrics

New Contributor

I am having a capacity scheduler setup with two queues - "low-priority", "regular-priority". There are two node-labels "low" and "regular". low-priority queue has 100% access to "low" node-label and regular-priority queue has 100% access to "regular" node label.

The capacity scheduler configuration(capacity-scheduler.xml):

<configuration>

<property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>1000</value>
</property>

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.2</value>
</property>

<property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
    <description>
        The ResourceCalculator implementation to be used to compare
        Resources in the scheduler.
        The default i.e. DefaultResourceCalculator only uses Memory while
        DominantResourceCalculator uses dominant-resource to compare
        multi-dimensional resources such as Memory, CPU etc.
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>low-priority,regular-priority</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels</name>
    <value>*</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels.regular.capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels.regular.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels.low.capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels.low.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
        The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>*</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
    <value>*</value>
    <description>
        The ACL of who can administer jobs on the default queue.
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
</property>

<property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.capacity</name>
    <value>50</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.ordering-policy</name>
    <value>fair</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.accessible-node-labels</name>
    <value>low</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.default-node-label-expression</name>
    <value>low</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.accessible-node-labels.low.capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.accessible-node-labels.low.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.default.state</name>
    <value>RUNNING</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.capacity</name>
    <value>50</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.ordering-policy</name>
    <value>fair</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.accessible-node-labels</name>
    <value>regular</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.default-node-label-expression</name>
    <value>regular</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.accessible-node-labels.regular.capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.accessible-node-labels.regular.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.default.state</name>
    <value>RUNNING</value>
</property>

Have attached the yarn ui capacity scheduler configuration. yarn ui scheduler When i see the QueueMetrics emitted by queue "low-priority" and "regular-priority" in (rm-ip:port/jmx), then it shows correct values of availableMB and availableVCores but when I submit a job to any queue, there is no update in jmx metrics like pendingMB, pendingVcores, availableMB, availableVCores etc. only AppsRunning, ActiveApplications etc. are getting updated. Not able to find why the metrics is not getting updated after job submission.

The issue comes only when node-label is enabled. When node-label is disabled and only queue is used everything works fine.

Don't have an account?
Coming from Hortonworks? Activate your account here