How to monitor the actual memory allocation of a spark application

[#1]

Is there a proper way to monitor the memory usage of a spark application.

By memory usage, i didnt mean the executor memory, that can be set, but the actual memory usage of the application.

Note : We are running Spark on YARN

[#2]

@Jonathan Sneep

I am not able to reply to your comment. Seems like the Reply option is not available.

Hence replying here

Comment

HDFS Write bytes by executor should look something like this (be sure to set the left Y unit type to bytes);

aliasByNode($application.*.executor.filesystem.*.write_bytes, 1)

Executor and Driver memory usage example (similarly as above set the left Y unit to bytes);

aliasByNode($application.*.jvm.heap.used, 1)

I'll try to find time later to give you some more examples, but they are mostly slight variations on the examples above : - )

Thanks for the comment. Will try the metrics queries and will let you know.

Also looking forward to your updates on the below as well

• HDFS Bytes Read/Written Per Executor
• HDFS Executor Read Bytes/Sec
• Read IOPS
• Task Executor
  
  • Active Tasks per Executor
  • Completed Tasks per Executor
  • Completed Tasks/Minute per Executor
• Driver Memory
  
  • Driver Heap Usage
  • Driver JVM Memory Pools Usage
  • Executor Memory Usage
  • JVM Heap Usage Per Executor

[#3]

@Jonathan Sneep
Could you please check if the below metrics queries are correct :

• Driver Memory
  
  • Driver Heap Usage
    aliasByNode($application.driver.jvm.heap.usage, 1)
  • Driver JVM Memory Pools Usage
    aliasByNode($application.driver.jvm.pools.*.used, 4)
• Executor & Driver Memory Used
  aliasByNode($application.*.jvm.heap.used, 1)
• Executor Memory Used
  aliasByNode(exclude($application.*.jvm.heap.used, '.driver.jvm.heap'), 1)
  alias(sumSeries(exclude($application.*.jvm.heap.used, '.driver.jvm.heap')), 'total')
• Task Executor
  • Active Tasks Per Executor
    aliasByNode(summarize($application.*.executor.threadpool.activeTasks, '10s', 'sum', false), 1)
  • Completed Tasks per Executor
    aliasByNode($application.*.executor.threadpool.completeTasks, 1)
  • Completed Tasks/Minute per Executor
    aliasByNode(nonNegativeDerivative(summarize($application.*.executor.threadpool.completeTasks, '1m', 'avg', false)), 1)
• Read/Write IOPS
  • Read IOPS
    alias(perSecond(sumSeries($application.*.executor.filesystem.hdfs.read_ops)), 'total')
    aliasByNode(perSecond($application.*.executor.filesystem.hdfs.read_ops), 1)
  • Write IOPS
    alias(perSecond(sumSeries($application.*.executor.filesystem.hdfs.write_ops)), 'total')
    aliasByNode(perSecond($application.*.executor.filesystem.hdfs.write_ops), 1)
• HDFS Bytes Reads/Writes Per Executor
  • Executor HDFS Reads
    aliasByMetric($application.*.executor.filesystem.hdfs.read_bytes)
  • Executor HDFS Bytes Written
    aliasByMetric($application.*.executor.filesystem.hdfs.write_bytes)

Also please let me know the queries for the below :

• HDFS Read/Write Byte Rate
  
  • HDFS Read Rate/Sec
  • HDFS Write Rate/Sec

Looking forward to your update regarding the same.

[#4]

@Jonathan Sneep

Did you had a chance to look into it.

[#5]

@Nikhil

Nice work. HDFS Write bytes by executor should look something like this (be sure to set the left Y unit type to bytes);

aliasByNode($application.*.executor.filesystem.*.write_bytes, 1)

Executor and Driver memory usage example (similarly as above set the left Y unit to bytes);

aliasByNode($application.*.jvm.heap.used, 1)

I'll try to find time later to give you some more examples, but they are mostly slight variations on the examples above : - )

[#6]

@Jonathan Sneep
Could you please check if the below metrics queries are correct :

• Driver Memory
  • Driver Heap Usage
    aliasByNode($application.driver.jvm.heap.usage, 1)
  • Driver JVM Memory Pools Usage
    aliasByNode($application.driver.jvm.pools.*.used, 4)
• Executor & Driver Memory Used
  aliasByNode($application.*.jvm.heap.used, 1)
• Executor Memory Used
  aliasByNode(exclude($application.*.jvm.heap.used, '.driver.jvm.heap'), 1)
  alias(sumSeries(exclude($application.*.jvm.heap.used, '.driver.jvm.heap')), 'total')
• Task Executor
  • Active Tasks Per Executor
    aliasByNode(summarize($application.*.executor.threadpool.activeTasks, '10s', 'sum', false), 1)
  • Completed Tasks per Executor
    aliasByNode($application.*.executor.threadpool.completeTasks, 1)
  • Completed Tasks/Minute per Executor
    aliasByNode(nonNegativeDerivative(summarize($application.*.executor.threadpool.completeTasks, '1m', 'avg', false)), 1)
• Read/Write IOPS
  • Read IOPS
    alias(perSecond(sumSeries($application.*.executor.filesystem.hdfs.read_ops)), 'total')
    aliasByNode(perSecond($application.*.executor.filesystem.hdfs.read_ops), 1)
  • Write IOPS
    alias(perSecond(sumSeries($application.*.executor.filesystem.hdfs.write_ops)), 'total')
    aliasByNode(perSecond($application.*.executor.filesystem.hdfs.write_ops), 1)
• HDFS Bytes Reads/Writes Per Executor
  • Executor HDFS Reads
    aliasByMetric($application.*.executor.filesystem.hdfs.read_bytes)
  • Executor HDFS Bytes Written
    aliasByMetric($application.*.executor.filesystem.hdfs.write_bytes)

Also please let me know the queries for the below :

• HDFS Read/Write Byte Rate
  • HDFS Read Rate/Sec
  • HDFS Write Rate/Sec

Looking forward to your update regarding the same.