HDP 2.3 - Calculate cpu and memory usage of completed hive Tez and map reduce jobs

[#1]

In HDP 2.5 you can use SmartSense Activity Reports to calculate the memory and cpu use for a query or map reduce job. Is there a way to do this in HDP 2.3? To calculate the amount of memory and CPU consumed by each query by mining the logs?

Looking in HDP 2.5 yarn logs I see several Final Counter entries (see example below) with cpu milliseconds, physical and virtual memory. Do I add up the values for all the Final Counter entries to get the total for the application?

yarn logs -applicationId application_1484323083293_0008 | grep -i memory | grep "Final Counters"

2017-01-13 19:05:58,515 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1484323083293_0008_3_01_000000_0: Counters: 40 [[File System Counte

rs FILE_BYTES_READ=0, FILE_BYTES_WRITTEN=7226, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=1134, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_

READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=0, GC_TIME_MILLIS=90, CPU_MILLISECONDS=3610, PHYSICAL_MEMORY_BYTES=315097088, VIRTUAL_ME

MORY_BYTES=2007277568,

[#2]

@cduby

You can identify the counters from Tez view, refer to link.

[#3]

I would like to do it programmatically to create a chargeback report.

[#4]

hi, bro have you finished your program? I am facing the same issue like yours

[#5]

@cduby

You can use tez apis for the same. Refer link.