Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

List all created spark jobs

avatar
Contributor

Hi,

I wanted to list all the spark jobs created. I tried looking in Resource Manager and Spark Job History Server and only active/failed/killed jobs are present. Is there a way to list all the spark jobs either running or not running? Could be in spark shell or any other that you can suggest.

Thanks.

1 ACCEPTED SOLUTION

avatar

The Spark History Server UI has a link at the bottom called "Show Incomplete Applications". Click on this link and it will show you the running jobs, like zeppelin (see image).

4453-sparkhistoryserverui-incompleteapps.png

View solution in original post

7 REPLIES 7

avatar
Guru

The Spark History server will have a list of all Jobs that have run using the YARN master.

If you are looking for current running jobs, the RM will give you a full list, though this will of course also include non-spark jobs running on your cluster.

If you are running spark standalone, you will not have any means of listing jobs.

avatar
Contributor

Thanks @Simon Elliston Ball. But is there a way to see created spark jobs that has not been deployed yet?

avatar

The Spark History Server UI has a link at the bottom called "Show Incomplete Applications". Click on this link and it will show you the running jobs, like zeppelin (see image).

4453-sparkhistoryserverui-incompleteapps.png

avatar
Contributor

Thanks @Paul Hargis. Yes, I tried it also but what I wanted to see is the created spark jobs that has not yet been deployed. Is there a way to find this jobs?

avatar
Cloudera Employee

Hi,

 

Could you please let us know what do you mean by not yet deployed? Does you mean that the jobs that has not been kicked off after you running the spark submit command (or) Could you please explain in detail.

 

 

Thanks

Akr

avatar

If you are running with deploy mode = yarn (previously, master set to "yarn-client" or "yarn-cluster"), then you can discover the state of the spark job by bringing up the Yarn ResourceManager UI. In Ambari, select Yarn service from left-hand panel, choose "Quick Links", and click on "ResourceManager UI". It will open web page on port 8088.

Here is an example (click on 'Applications' in left panel to see all states):

4494-yarn-resourcemanager.png

avatar
Contributor
@Paul Hargis

Thanks for the info, appreciate your help on this.