Spark provides several ways to expose metrics (http://spark.apache.org/docs/latest/monitoring.html). Spark provides a GraphiteSink for sending metrics to Graphite. Unfortunately there's no native way to send metrics to Prometheus. To use Prometheus you scrape from Graphite like suggested it the blog. Another way is to expose Spark metrics via the JmxSink, and set up a Prometheus JMX exporter to transform the JMX MBeans into metrics readable by Prometheus. A nice blog about it is posted here: https://argus-sec.com/blog/monitoring-spark-prometheus.
... View more
