Created 01-13-2017 07:40 PM
In HDP 2.5 you can use SmartSense Activity Reports to calculate the memory and cpu use for a query or map reduce job. Is there a way to do this in HDP 2.3? To calculate the amount of memory and CPU consumed by each query by mining the logs?
Looking in HDP 2.5 yarn logs I see several Final Counter entries (see example below) with cpu milliseconds, physical and virtual memory. Do I add up the values for all the Final Counter entries to get the total for the application?
yarn logs -applicationId application_1484323083293_0008 | grep -i memory | grep "Final Counters"
2017-01-13 19:05:58,515 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1484323083293_0008_3_01_000000_0: Counters: 40 [[File System Counte
rs FILE_BYTES_READ=0, FILE_BYTES_WRITTEN=7226, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=1134, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_
READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=0, GC_TIME_MILLIS=90, CPU_MILLISECONDS=3610, PHYSICAL_MEMORY_BYTES=315097088, VIRTUAL_ME
MORY_BYTES=2007277568,
Created 01-13-2017 08:54 PM
Created 01-17-2017 01:11 AM
I would like to do it programmatically to create a chargeback report.
Created 04-20-2017 02:49 AM
hi, bro have you finished your program? I am facing the same issue like yours
Created 01-17-2017 06:42 PM