Is it possible to get the cpu utilization of spark job programmatically using the application Id?

[#1]

Hi all,

I would like to know whether we can fetch the cpu utilization of a spark job programmatically using the application Id. Any help would be appreciated.

Thank you

[#2]

Hello @KSKR ,

thank you for raising the question on "how to fetch the CPU utilization for a Spark job programmatically".

One way to do this is via the Spark REST API.

You should consider if you need the "live data" or you are looking for analysis once the application finished running.

While the application is running, you can consider to connect to the driver and fetch the live data. Once the application finished running, you can consider parse the JSON files (the event log files) for the CPU time or use the Spark REST API and let the Spark History Server serve you with the data.

What is your exact requirement? What would you like to achieve?

Thank you:

Ferenc