Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark metrics sink doesn't expose executor's metrics

avatar
Explorer

I'm using Spark on YARN with

Ambari 2.7.4
HDP Standalone 3.1.4
Spark 2.3.2
Hadoop 3.1.1
Graphite on Docker latest

I was trying to get Spark metrics with Graphite sink following this tutorial.

Advanced spark2-metrics-properties in Ambari are:

driver.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
executor.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
worker.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
master.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
*.sink.graphite.host=ap-test-m.c.gcp-ps.internal
*.sink.graphite.port=2003
*.sink.graphite.protocol=tcp
*.sink.graphite.period=10
*.sink.graphite.unit=seconds
*.sink.graphite.prefix=app-test
*.source.jvm.class=org.apache.spark.metrics.source.JvmSource

Spark submit:

export HADOOP_CONF_DIR=/usr/hdp/3.1.4.0-315/hadoop/conf/; spark-submit --class com.Main --master yarn --deploy-mode client --driver-memory 1g --executor-memory 10g --num-executors 2 --executor-cores 2 spark-app.jar /data

As a result I'm only getting driver metrics.

Also, I was trying to add metrics.properties to spark-submit command together with global spark metrics props, but that didn't help.

And finally, I tried conf in spark-submit and in java SparkConf:

--conf "spark.metrics.conf.driver.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" 
--conf "spark.metrics.conf.executor.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" 
--conf "worker.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink"
--conf "master.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink"
--conf "spark.metrics.conf.*.sink.graphite.host"="host" 
--conf "spark.metrics.conf.*.sink.graphite.port"=2003 
--conf "spark.metrics.conf.*.sink.graphite.period"=10 
--conf "spark.metrics.conf.*.sink.graphite.unit"=seconds 
--conf "spark.metrics.conf.*.sink.graphite.prefix"="app-test" 
--conf "spark.metrics.conf.*.source.jvm.class"="org.apache.spark.metrics.source.JvmSource"

But that didn't help either.
I was trying to submit jobs in both client and cluster modes.

CSVSink also gives only driver metrics.

3 REPLIES 3

avatar
Explorer

UPD

When I submit job in cluster mode - I'm getting the same metrics for executor as in Spark History Server. But the jvm metrics are still absent.

avatar
New Contributor

I am facing the same situation.

avatar
New Contributor

I am also facing same issue. Did anyone find solutions for this issue?