Created 11-03-2019 10:10 AM
I'm using Spark on YARN with
Ambari 2.7.4
HDP Standalone 3.1.4
Spark 2.3.2
Hadoop 3.1.1
Graphite on Docker latest
I was trying to get Spark metrics with Graphite sink following this tutorial.
Advanced spark2-metrics-properties in Ambari are:
driver.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink executor.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink worker.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink master.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink *.sink.graphite.host=ap-test-m.c.gcp-ps.internal *.sink.graphite.port=2003 *.sink.graphite.protocol=tcp *.sink.graphite.period=10 *.sink.graphite.unit=seconds *.sink.graphite.prefix=app-test *.source.jvm.class=org.apache.spark.metrics.source.JvmSource
Spark submit:
export HADOOP_CONF_DIR=/usr/hdp/3.1.4.0-315/hadoop/conf/; spark-submit --class com.Main --master yarn --deploy-mode client --driver-memory 1g --executor-memory 10g --num-executors 2 --executor-cores 2 spark-app.jar /data
As a result I'm only getting driver metrics.
Also, I was trying to add metrics.properties to spark-submit command together with global spark metrics props, but that didn't help.
And finally, I tried conf in spark-submit and in java SparkConf:
--conf "spark.metrics.conf.driver.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" --conf "spark.metrics.conf.executor.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" --conf "worker.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" --conf "master.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" --conf "spark.metrics.conf.*.sink.graphite.host"="host" --conf "spark.metrics.conf.*.sink.graphite.port"=2003 --conf "spark.metrics.conf.*.sink.graphite.period"=10 --conf "spark.metrics.conf.*.sink.graphite.unit"=seconds --conf "spark.metrics.conf.*.sink.graphite.prefix"="app-test" --conf "spark.metrics.conf.*.source.jvm.class"="org.apache.spark.metrics.source.JvmSource"
But that didn't help either.
I was trying to submit jobs in both client and cluster modes.
CSVSink also gives only driver metrics.
Created on 11-04-2019 03:23 AM - edited 11-04-2019 03:23 AM
UPD
When I submit job in cluster mode - I'm getting the same metrics for executor as in Spark History Server. But the jvm metrics are still absent.
Created 06-15-2020 01:48 AM
I am facing the same situation.
Created 12-29-2020 04:14 PM
I am also facing same issue. Did anyone find solutions for this issue?