check memory and cpu usage

SaF007 — Fri, 06 May 2022 11:47:38 GMT

there are like 1k finished jobs and i want to check allocated CPU and memory of those jobs. how to do that? i tried in command line but it shows when i enter job id and in my yarn UI CPU and memory metrics are not there.

Re: check memory and cpu usage

shehbazk — Wed, 11 May 2022 08:21:41 GMT

To get the number of vcores and memory used for a particular queue in the cluster 

You can get this information in the cluster utilization report, please go through the below doc for more information:
https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_cluster_util_report.html

Also, if you want to see the resources used for each application, please go through the below KB article:
https://my.cloudera.com/knowledge/How-to-calculate-memory-and-v-core-utilization-for-each-of-the?id=...

another option is
Using yarn application -status command, you can get the Aggregate Resource Allocation for an application.
For e.g. yarn application -status application_ID (for one of completed applications), one of the rows returned is:

Aggregate Resource Allocation : 46641 MB-seconds, 37 vcore-seconds

This gives an aggregate memory and CPU allocations in seconds.

https://my.cloudera.com/knowledge/How-to-calculate-memory-and-v-core-utilization-for-each-of-the?id=...

question check memory and cpu usage in Support Questions

check memory and cpu usage

Re: check memory and cpu usage