Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎04-05-2017

Having the Consumed and Allocated RAM per JOB / Cloudera Manager API

Hello guys!

 

I wonder if anyone can help me find the right metric to know how much RAM and CPU has been allocated for a job, and how much were really consumed.

 

I'm working with a Scala Program with the Cloudera API, here is a sample of code:

 

 

val apiRoot: RootResourceV11 = new ClouderaManagerClientBuilder()
        .withBaseURL(new URL("https://cm.bigdata.intraxa")).withHost("cm.bigdata.intraxa").withUsernamePassword("afakir", "loloflam")
      .enableTLS()
      .enableLogging()
      .build()
      .getRootV11()

    val clrV12 = apiRoot.getClustersResource
    val srV11: ServicesResourceV11 = clrV12.getServicesResource("cluster")
    val yar: YarnApplicationsResource = srV11.getYarnApplicationsResource("yarn2")//yarn / yarn2 ==> daas / bigplay
    val ayar  = yar.getYarnApplications("yarn2", "state=SUCCEEDED",DateTime.now().minusDays(7).secondOfDay().withMinimumValue().toString,DateTime.now().toString,1000,0)//DateTime.now().minusDays(1).secondOfDay().withMinimumValue().toString, DateTime.now().minusDays(1).secondOfDay().withMaximumValue().toString, 700, 0)

    val listOfApps = ayar.getApplications

In order th have, for example, the CPU consumed by a job, I run this:

 

 

 

listOfApps.get(0).getAttributes.get("cm_cpu_milliseconds")

In the API documentation (https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_dg_yarn_applications.html), I didn't find how I can have the consumed and allocated RAM for a job, does the consumed RAM refers to the Virtual Memory? Or the Physical Memory? Can you help me please make the differenciation with that?

 

Thank you so much, and any other documentation that might help is welcomed.

 

Kind Regards,

 

Ayoub

 

Announcements