04-05-2017 05:39 AM
I wonder if anyone can help me find the right metric to know how much RAM and CPU has been allocated for a job, and how much were really consumed.
I'm working with a Scala Program with the Cloudera API, here is a sample of code:
val apiRoot: RootResourceV11 = new ClouderaManagerClientBuilder() .withBaseURL(new URL("https://cm.bigdata.intraxa")).withHost("cm.bigdata.intraxa").withUsernamePassword("afakir", "loloflam") .enableTLS() .enableLogging() .build() .getRootV11() val clrV12 = apiRoot.getClustersResource val srV11: ServicesResourceV11 = clrV12.getServicesResource("cluster") val yar: YarnApplicationsResource = srV11.getYarnApplicationsResource("yarn2")//yarn / yarn2 ==> daas / bigplay val ayar = yar.getYarnApplications("yarn2", "state=SUCCEEDED",DateTime.now().minusDays(7).secondOfDay().withMinimumValue().toString,DateTime.now().toString,1000,0)//DateTime.now().minusDays(1).secondOfDay().withMinimumValue().toString, DateTime.now().minusDays(1).secondOfDay().withMaximumValue().toString, 700, 0) val listOfApps = ayar.getApplications
In order th have, for example, the CPU consumed by a job, I run this:
In the API documentation (https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_dg_yarn_applications.html), I didn't find how I can have the consumed and allocated RAM for a job, does the consumed RAM refers to the Virtual Memory? Or the Physical Memory? Can you help me please make the differenciation with that?
Thank you so much, and any other documentation that might help is welcomed.