@fil I would suggest you to create a shell script that pick this data from Yarn resource manager . I created for me a shell script and pick the data on a daily basis and aggregare the memory and CPU time for each pool, for sure you can do that per user and even job if needed. See below: Note: i grap some of the code which related to my data center and i may delete something that will result the script to fail in your side, try to play around. Let me know if you need any more help with that, for sure you can change here the queue with user. #!/bin/bash STARTDATE=`date -d " -1 day " +%s%N | cut -b1-13` ENDDATE=`date +%s%N | cut -b1-13` result=`curl -s http://yarn_resource_manager:8088/ws/v1/cluster/apps?finishedTimeBegin=$STARTDATE&finishedTimeEnd=$ENDDATE` if [[ $result =~ "standby RM" ]]; then result=`curl -s http://yarn_resource_manager2:8088/ws/v1/cluster/apps?finishedTimeBegin=$STARTDATE&finishedTimeEnd=$ENDDATE` echo $result | python -m json.tool | sed 's/["|,]//g' | grep -E "queue|coreSeconds" | awk -v ' /queue/ { queue = $2 } /vcoreSeconds/ { arr[queue]+=$2 ; } END { for (x in arr) {print ".yarn." x ".cpums="arr[x]} } ' echo $result | python -m json.tool | sed 's/["|,]//g' | grep -E "queue|memorySeconds" | awk ' /queue/ { queue = $2 } /memorySeconds/ { arr1[queue]+=$2 ; } END { for (y in arr1) {print ".yarn." y ".memorySeconds="arr1[y]} } '
... View more
