YARN - Is there a metric in RM for Number of Containers got preempted over time

[#1]

Hi,

I am looking for a Resource Manager metric where it emits metrics related to number of containers preempted, amount of preempted MB and Vcores.

I am able to get this metric from each application, but looking if resource manager can emit these metrics for each queue. I have searched through all the different metrics in AMS but it doesn't have it.

HDPv2.5.0 or Hadoopv2.7.3

[#2 · Accepted solution]

@Saikiran Parepally

I don't think queue level preemption metrics exists. However, they are fairly easy to calculate from the app level metrics.

curl http://YOUR_RM_ADDRESS.com:8088/ws/v1/cluster/apps > /tmp/apps
queues=$(cat /tmp/apps | jq '.apps.app[].queue' | sort -u)
for queue in $queues; do
    echo $queue
    metrics="preemptedResourceMB preemptedResourceVCores numNonAMContainerPreempted numAMContainerPreempted"
    for metric in $metrics; do
        printf "%30s: " $metric 
        cat /tmp/apps | jq -r ".apps.app[] | select(.queue == $queue) .$metric" | paste -s -d+ - | bc
    done    
done

Most likely there are more efficient ways to to do this calculation in higher level programming languages, or if you are a jq expert.

[#3]

Thanks @gnovak currently I am getting app level details similarly as mentioned above using REST API. I am interested to see if there is anything at queue level.