Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CPU and Memory Usage per job perspective.

CPU and Memory Usage per job perspective.

Explorer

Actually if we account any job using resource usage such as CPU and Memory,So which metrics we need to check allocated Vcore seconds or CPU time and same as for Ram usage like allocated memory seconds or physical memory .

 

This is due to that at any instant what is the total CPU and RAM usage from the jobs running across the cluster.

2 REPLIES 2
Highlighted

Re: CPU and Memory Usage per job perspective.

Contributor

Hi Nickk,

 

If you are looking for what features that are available for YARN resource accounting, we do have two metrics available within the YARN API, as well as a more robust reporting capability within Cloudera Manager 5.7 onward.

 

The following are the definitions of memorySeconds and vcoreSeconds which are used to provide a very basic measurement of utilization in YARN[1]:


memorySeconds = The aggregated amount of memory (in megabytes) the application has allocated times the number of seconds the application has been running.

vcoreSeconds = The aggregated number of vcores that the application has allocated times the number of seconds the application has been running.


The memorySeconds value can be used loosely for generically measuring the amount of resource that the job consumed; For example, job 1 used X amount of memorySeconds as compared to job 2 which used Y amount of memorySeconds. Any further calculations attempting to extrapolate further insight from this measure isn't recommended.

There are some additional reporting efforts that are being worked on, one is now available with CM. Starting in CM 5.7 on, CM offers cluster utilization reporting which can help provide per tenant/user cluster usage reporting.  Further details regarding Cluster Utilization reporting in CM is available here[2]. 


References:
[1] Link to ApplicationResourceUsageReport.java (part of the YARN API) in the Apache source code for Hadoop:  
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main...

[2] Link to Cloudera Documentation regarding CM's Cluster Utilization Reporting functionality:
http://www.cloudera.com/documentation/enterprise/5-7-x/topics/admin_cluster_util_report.html

 

 

Hope this helps!

 

 

Re: CPU and Memory Usage per job perspective.

New Contributor

Hi,

     For most the services, roles, etc,  I see that the CPU utilization and other metrics obtained are at the minimum minute granularity, can someone please let me know how do I get at seconds or milliseconds granularity. I appreciate your response. 

 

Regards

Harsha

Don't have an account?
Coming from Hortonworks? Activate your account here