A good starting point is to review mistake number 1, in the slideshare: https://www.slideshare.net/SparkSummit/top-5-mistakes-when-writing-spark-applications-63071421 This gives a good starting point in tuning the cores, executor memory, etc.   ... View more

You are requesting how to get the "Per job" memory and cpu counters.   Please see the recent response in:   https://community.cloudera.com/t5/Support-Questions/How-to-get-the-YARN-jobs-metadata-directly-not-using-API/m-p/322711/highlight/false#M228910   In the metadata (counter) output, you will see the vcore-milliseconds and vcore-millseconds value for all map and reduce tasks, Task  Summary,  Analysis, File System  Counters for the job and other info about the specific job. ... View more

You have the following options to see the jobs counters (metadata). mapred job -history /usr/history/done/<date>/<job>.jhist -format human|json For the json format, you can pipe its output to python -m json.tool for a cleaner output. Note that JHS seeds the jobs from .jhist files (For every job, there is one .jhist file) that are stored in the HDFS directory, by default /user/history/done. The .jhist files are generated by individual job before the job completes. You may access this metadata from the .jhist files with the above commands.  If the AM failed to move its .jhist file to the directory that JHS looks for, JHS has no idea of the job at all. mapred job -history /usr/history/done/<date>/<job>.jhist -format human mapred job -history /usr/history/done/<date>/<job>.jhist -format json mapred job -history /usr/history/done/<date>/<job>.jhist -format json | python -m json.tool  No password is needed, but you might have to kinit if in a kerberized environment.  ... View more