Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is it possible to get the cpu utilization of spark job programmatically using the application Id?

Highlighted

Is it possible to get the cpu utilization of spark job programmatically using the application Id?

New Contributor

Hi all,

I would like to know whether we can fetch the cpu utilization of a spark job programmatically using the application Id. Any help would be appreciated.

Thank you

1 REPLY 1

Re: Is it possible to get the cpu utilization of spark job programmatically using the application Id?

Moderator

Hello @KSKR ,

 

thank you for raising the question on "how to fetch the CPU utilization for a Spark job programmatically".

 

One way to do this is via the Spark REST API.

You should consider if you need the "live data" or you are looking for analysis once the application finished running.

While the application is running, you can consider to connect to the driver and fetch the live data. Once the application finished running, you can consider parse the JSON files (the event log files) for the CPU time or use the Spark REST API and let the Spark History Server serve you with the data.

 

What is your exact requirement? What would you like to achieve?

 

Thank you:

Ferenc

 

 


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Don't have an account?
Coming from Hortonworks? Activate your account here