question Re: Measuring Spark job performance in Support Questions

question Re: Measuring Spark job performance in Support Questions https://community.cloudera.com/t5/Support-Questions/Measuring-Spark-job-performance/m-p/308299#M223509 Hi <a href="https://community.cloudera.com/t5/user/viewprofilepage/user-id/83689">@TimmehG</a>, Spark has a configurable metrics system based on the <A href="http://metrics.dropwizard.io/4.1.1" target="_blank" rel="noopener">Dropwizard Metrics Library</A>. This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV files. The metrics are generated by sources embedded in the Spark codebase. They provide instrumentation for specific activities and Spark components. The metrics system is configured via a configuration file that Spark expects to be present at $SPARK_HOME/conf/metrics.properties. A custom file location can be specified via the spark.metrics.conf <A href="https://spark.apache.org/docs/latest/configuration.html#spark-properties" target="_blank" rel="noopener">configuration property</A>. Instead of using the configuration file, a set of configuration parameters with prefix spark.metrics.conf. can be used.  I agree with you, running spark applications continuously & reliably is a challenging task, and a good performance monitoring system is needed. Several external tools can be used to help profile the performance of Spark jobs:<UL><LI>Cluster-wide monitoring tools, such as <A href="http://ganglia.sourceforge.net/" target="_blank" rel="noopener">Ganglia</A>, can provide insight into overall cluster utilization and resource bottlenecks. For instance, a Ganglia dashboard can quickly reveal whether a particular workload is disk-bound, network bound, or CPU bound.</LI><LI>OS profiling tools such as <A href="http://dag.wieers.com/home-made/dstat/" target="_blank" rel="noopener">dstat</A>, <A href="http://linux.die.net/man/1/iostat" target="_blank" rel="noopener">iostat</A>, and <A href="http://linux.die.net/man/1/iotop" target="_blank" rel="noopener">iotop</A> can provide fine-grained profiling on individual nodes.</LI><LI>JVM utilities such as jstack for providing stack traces, jmap for creating heap-dumps, jstat for reporting time-series statistics and jconsole for visually exploring various JVM properties are useful for those comfortable with JVM internals.</LI></UL>For more insights you can refer to the below links:<A href="https://spark.apache.org/docs/latest/monitoring.html" target="_blank" rel="noopener">https://spark.apache.org/docs/latest/monitoring.html</A><A href="https://blog.cloudera.com/demystifying-spark-jobs-to-optimize-for-cost-and-performance/" target="_blank" rel="noopener">https://blog.cloudera.com/demystifying-spark-jobs-to-optimize-for-cost-and-performance/</A><A href="https://www.infoq.com/articles/spark-application-monitoring-influxdb-grafana/" target="_blank" rel="noopener">https://www.infoq.com/articles/spark-application-monitoring-influxdb-grafana/</A><A href="https://db-blog.web.cern.ch/blog/luca-canali/2017-03-measuring-apache-spark-workload-metrics-performance-troubleshooting" target="_blank" rel="noopener">https://db-blog.web.cern.ch/blog/luca-canali/2017-03-measuring-apache-spark-workload-metrics-performance-troubleshooting</A> Please accept the answer you found most useful. Wed, 23 Dec 2020 06:15:48 GMT jagadeesan 2020-12-23T06:15:48Z