Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Having the Consumed and Allocated RAM per JOB / Cloudera Manager API

Highlighted

Having the Consumed and Allocated RAM per JOB / Cloudera Manager API

New Contributor

Hello guys!

 

I wonder if anyone can help me find the right metric to know how much RAM and CPU has been allocated for a job, and how much were really consumed.

 

I'm working with a Scala Program with the Cloudera API, here is a sample of code:

 

 

val apiRoot: RootResourceV11 = new ClouderaManagerClientBuilder()
        .withBaseURL(new URL("https://cm.bigdata.intraxa")).withHost("cm.bigdata.intraxa").withUsernamePassword("afakir", "loloflam")
      .enableTLS()
      .enableLogging()
      .build()
      .getRootV11()

    val clrV12 = apiRoot.getClustersResource
    val srV11: ServicesResourceV11 = clrV12.getServicesResource("cluster")
    val yar: YarnApplicationsResource = srV11.getYarnApplicationsResource("yarn2")//yarn / yarn2 ==> daas / bigplay
    val ayar  = yar.getYarnApplications("yarn2", "state=SUCCEEDED",DateTime.now().minusDays(7).secondOfDay().withMinimumValue().toString,DateTime.now().toString,1000,0)//DateTime.now().minusDays(1).secondOfDay().withMinimumValue().toString, DateTime.now().minusDays(1).secondOfDay().withMaximumValue().toString, 700, 0)

    val listOfApps = ayar.getApplications

In order th have, for example, the CPU consumed by a job, I run this:

 

 

 

listOfApps.get(0).getAttributes.get("cm_cpu_milliseconds")

In the API documentation (https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_dg_yarn_applications.html), I didn't find how I can have the consumed and allocated RAM for a job, does the consumed RAM refers to the Virtual Memory? Or the Physical Memory? Can you help me please make the differenciation with that?

 

Thank you so much, and any other documentation that might help is welcomed.

 

Kind Regards,

 

Ayoub