https://community.cloudera.com/t5/Support-Questions/Measuring-Spark-job-performance/m-p/307178#M223142 <P>I have a Spark job where I am tuning the code and settings to improve performance. However I'm not sure how to track the performance when using a shared cluster to see if I'm making things better or worse!</P><P>&nbsp;</P><P>I've considered the following metrics, though with each of them even when running an identical job twice I can get very different values, so it doesn't seem a good way to track the performance of the job.</P><UL><LI>Aggregated resource usage (CPU and Memory)</LI><LI>Task time (active and total)</LI><LI>Job run time</LI></UL><P>Ideally I'd like to be able to track the performance of the job over time as the codebase evolves</P> Tue, 08 Dec 2020 17:00:26 GMT TimmehG 2020-12-08T17:00:26Z