How do find Aggregate Resource Allocation for an application run on Yarn ?
I know Yarn application -status <application Id> provides that info, but it doesn't work for historical jobs as they won't present in Resource Manager then. I see I could find this information from Cloudera Manager though. Is Cloudera Manager preserving this information upon their expiry ? if so, Is there any API that I can leverage to retrieve this info for historical jobs.
Alternatively, I glanced through the job history server counters but couldn't tie out while reconciling which gives me a strange feeling that I might be looking at wrong counters.
The information I am looking for is highlighted below
Thanks for sharing the API, I looked at this before. The problem with this API is, it will only retrieve application stastics that is still available in RM. I reckon, application that are older than 24-48 hours are moved out of RM to JHIST server. So those application stats are no longer retrievable. I can do batch collection everyday and store them locally for retrieving later, but I am keen on understanding how Cloudera Manager is retrieving this info i.e. via some API or logs from cluster or uses its own internal database with frequent backend stats collection.
Makes sense, I wanted to do the same... Like extract and load into db for later retrieval but was wondering if they were stored historically anywhere. Thanks Fawze. I will accept this as answer if there is no other way to retrieve.
Another question on the same lines, if you don't mind answering. What is the criteria to calculate the aggregate resource allocation, meaning, is it possible to arrive at those figures by using any of the counters in Job History server for the applications ?
@Fawze We have both but majority of them are MapReduce. They all run on YARN though. I could use Spark history to collect metrics for that but for MapReduce, still trying.
I tried summation of Vcores seconds taken my Mapper & Reducer, yet they don't tally with the Aggregated value I see in Cloudera Manager or in Yarn Application -status <app_id>. Perhaps, I am overlooking something during conversion. I am doing Milliseconds to seconds.. Hope, I am doing it right