Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Finding Yarn Aggregate Resource Allocation

Solved Go to solution

Finding Yarn Aggregate Resource Allocation

Explorer

Hi,

 

How do find Aggregate Resource Allocation for an application run on Yarn ? 

 

I know Yarn application -status <application Id> provides that info, but it doesn't work for historical jobs as they won't present in Resource Manager then. I see I could find this information from Cloudera Manager though. Is Cloudera Manager preserving this information upon their expiry ? if so, Is there any API that I can leverage to retrieve this info for historical jobs. 

 

Alternatively, I glanced through the job history server counters but couldn't tie out while reconciling which gives me a strange feeling that I might be looking at wrong counters. 

 

The information I am looking for is highlighted below 

 

yarn application -status application_XXXXXXXXXXX_XXX97

 

Application-Id : application_XXXXXXXXXXX_XXX97
Application-Name : blah blah blah
Application-Type : MAPREDUCE
User : user1
Queue : queue1
Start-Time : 1494279901683
Finish-Time : 1494280086823
Progress : 100%
State : FINISHED
Final-State : SUCCEEDED
Aggregate Resource Allocation : 674854 MB-seconds, 329 vcore-seconds
Log Aggregation Status : SUCCEEDED
Diagnostics :

 

Interested in either using an API if this information is readily available or how do I calculate it from the available data in JOb history counters. 

 

Thanks in advance

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Finding Yarn Aggregate Resource Allocation

Super Collaborator
You can increase the time series to store more historical data.

What i did in my side, i write a script that collect the data daily and
populate it to graphana.
9 REPLIES 9

Re: Finding Yarn Aggregate Resource Allocation

Super Collaborator

Hi @code0404 Yes, you can use the Yarn histort API

 

Please see the following documntation:

 

https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Re: Finding Yarn Aggregate Resource Allocation

Explorer

@Fawze wrote:

Hi @code0404 Yes, you can use the Yarn histort API

 

Please see the following documntation:

 

https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html


Hi @Fawze

 

Thanks for sharing the API, I looked at this before. The problem with this API is, it will only retrieve application stastics that is still available in RM. I reckon, application that are older than 24-48 hours are moved out of RM to JHIST server. So those application stats are no longer retrievable. I can do batch collection everyday and store them locally for retrieving later, but I am keen on understanding how Cloudera Manager is retrieving this info i.e. via some API or logs from cluster or uses its own internal database with frequent backend stats collection. 

Re: Finding Yarn Aggregate Resource Allocation

Super Collaborator
You can increase the time series to store more historical data.

What i did in my side, i write a script that collect the data daily and
populate it to graphana.

Re: Finding Yarn Aggregate Resource Allocation

Super Collaborator
You can increase the time series to store more historical data.

What i did in my side, i write a script that collect the data daily and
populate it to graphana.

Re: Finding Yarn Aggregate Resource Allocation

Explorer

Makes sense, I wanted to do the same... Like extract and load into db for later retrieval but was wondering if they were stored historically anywhere. Thanks Fawze. I will accept this as answer if there is no other way to retrieve. 

Re: Finding Yarn Aggregate Resource Allocation

Explorer

Another question on the same lines, if you don't mind answering. What is the criteria to calculate the aggregate resource allocation, meaning, is it possible to arrive at those figures by using any of the counters in Job History server for the applications ? 

Highlighted

Re: Finding Yarn Aggregate Resource Allocation

Super Collaborator

@code0404 Do you have spark jobs or all your jobs are mapreduce?

 

normally summation of vcores and memory seconds are enough.

Re: Finding Yarn Aggregate Resource Allocation

Explorer

@Fawze We have both but majority of them are MapReduce. They all run on YARN though. I could use Spark history to collect metrics for that but for MapReduce, still trying. 

 

I tried summation of Vcores seconds taken my Mapper & Reducer, yet they don't tally with the Aggregated value I see in Cloudera Manager or in Yarn Application -status <app_id>. Perhaps, I am overlooking something during conversion. I am doing Milliseconds to seconds.. Hope, I am doing it right

Re: Finding Yarn Aggregate Resource Allocation

Super Collaborator

@code0404 

See how i'm doing this, but i collect the aggregate metrics per pool, you can just use by the application name or the user.

 

STARTDATE=`date -d " -1 day " +%s%N | cut -b1-13`
ENDDATE=`date +%s%N | cut -b1-13`

result=`curl -s http://your-yarn-history-server:8088/ws/v1/cluster/apps?finishedTimeBegin=$STARTDATE&finishedTimeEnd=$ENDDATE`

echo $result | python -m json.tool | sed 's/["|,]//g' | grep -E "queue|coreSeconds" | awk -v DC="$DC" ' /queue/ { queue = $2 }
/vcoreSeconds/ { arr[queue]+=$2 ; }
END { for (x in arr) {print DC ".yarn." x ".cpums="arr[x]} } '

echo $result | python -m json.tool | sed 's/["|,]//g' | grep -E "queue|memorySeconds" | awk -v DC="$DC" ' /queue/ { queue = $2 }
/memorySeconds/ { arr1[queue]+=$2 ; }
END { for (y in arr1) {print DC ".yarn." y ".memorySeconds="arr1[y]} } '