Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP 2.3 - Calculate cpu and memory usage of completed hive Tez and map reduce jobs

Highlighted

HDP 2.3 - Calculate cpu and memory usage of completed hive Tez and map reduce jobs

In HDP 2.5 you can use SmartSense Activity Reports to calculate the memory and cpu use for a query or map reduce job. Is there a way to do this in HDP 2.3? To calculate the amount of memory and CPU consumed by each query by mining the logs?

Looking in HDP 2.5 yarn logs I see several Final Counter entries (see example below) with cpu milliseconds, physical and virtual memory. Do I add up the values for all the Final Counter entries to get the total for the application?

yarn logs -applicationId application_1484323083293_0008 | grep -i memory | grep "Final Counters"

2017-01-13 19:05:58,515 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1484323083293_0008_3_01_000000_0: Counters: 40 [[File System Counte

rs FILE_BYTES_READ=0, FILE_BYTES_WRITTEN=7226, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=1134, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_

READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=0, GC_TIME_MILLIS=90, CPU_MILLISECONDS=3610, PHYSICAL_MEMORY_BYTES=315097088, VIRTUAL_ME

MORY_BYTES=2007277568,

4 REPLIES 4

Re: HDP 2.3 - Calculate cpu and memory usage of completed hive Tez and map reduce jobs

@cduby

You can identify the counters from Tez view, refer to link.

Re: HDP 2.3 - Calculate cpu and memory usage of completed hive Tez and map reduce jobs

I would like to do it programmatically to create a chargeback report.

Re: HDP 2.3 - Calculate cpu and memory usage of completed hive Tez and map reduce jobs

New Contributor

hi, bro have you finished your program? I am facing the same issue like yours

Re: HDP 2.3 - Calculate cpu and memory usage of completed hive Tez and map reduce jobs

@cduby

You can use tez apis for the same. Refer link.

Don't have an account?
Coming from Hortonworks? Activate your account here