- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Spark metrics sink doesn't expose executor's metrics
Created ‎11-03-2019 10:10 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm using Spark on YARN with
Ambari 2.7.4
HDP Standalone 3.1.4
Spark 2.3.2
Hadoop 3.1.1
Graphite on Docker latest
I was trying to get Spark metrics with Graphite sink following this tutorial.
Advanced spark2-metrics-properties in Ambari are:
driver.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink executor.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink worker.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink master.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink *.sink.graphite.host=ap-test-m.c.gcp-ps.internal *.sink.graphite.port=2003 *.sink.graphite.protocol=tcp *.sink.graphite.period=10 *.sink.graphite.unit=seconds *.sink.graphite.prefix=app-test *.source.jvm.class=org.apache.spark.metrics.source.JvmSource
Spark submit:
export HADOOP_CONF_DIR=/usr/hdp/3.1.4.0-315/hadoop/conf/; spark-submit --class com.Main --master yarn --deploy-mode client --driver-memory 1g --executor-memory 10g --num-executors 2 --executor-cores 2 spark-app.jar /data
As a result I'm only getting driver metrics.
Also, I was trying to add metrics.properties to spark-submit command together with global spark metrics props, but that didn't help.
And finally, I tried conf in spark-submit and in java SparkConf:
--conf "spark.metrics.conf.driver.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" --conf "spark.metrics.conf.executor.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" --conf "worker.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" --conf "master.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" --conf "spark.metrics.conf.*.sink.graphite.host"="host" --conf "spark.metrics.conf.*.sink.graphite.port"=2003 --conf "spark.metrics.conf.*.sink.graphite.period"=10 --conf "spark.metrics.conf.*.sink.graphite.unit"=seconds --conf "spark.metrics.conf.*.sink.graphite.prefix"="app-test" --conf "spark.metrics.conf.*.source.jvm.class"="org.apache.spark.metrics.source.JvmSource"
But that didn't help either.
I was trying to submit jobs in both client and cluster modes.
CSVSink also gives only driver metrics.
Created on ‎11-04-2019 03:23 AM - edited ‎11-04-2019 03:23 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
UPD
When I submit job in cluster mode - I'm getting the same metrics for executor as in Spark History Server. But the jvm metrics are still absent.
Created ‎06-15-2020 01:48 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am facing the same situation.
Created ‎12-29-2020 04:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am also facing same issue. Did anyone find solutions for this issue?
