List all created spark jobs

[#1]

Hi,

I wanted to list all the spark jobs created. I tried looking in Resource Manager and Spark Job History Server and only active/failed/killed jobs are present. Is there a way to list all the spark jobs either running or not running? Could be in spark shell or any other that you can suggest.

Thanks.

[#2]

The Spark History server will have a list of all Jobs that have run using the YARN master.

If you are looking for current running jobs, the RM will give you a full list, though this will of course also include non-spark jobs running on your cluster.

If you are running spark standalone, you will not have any means of listing jobs.

[#3]

Thanks @Simon Elliston Ball. But is there a way to see created spark jobs that has not been deployed yet?

[#4 · Accepted solution]

The Spark History Server UI has a link at the bottom called "Show Incomplete Applications". Click on this link and it will show you the running jobs, like zeppelin (see image).

[#5]

Thanks @Paul Hargis. Yes, I tried it also but what I wanted to see is the created spark jobs that has not yet been deployed. Is there a way to find this jobs?

[#6]

Hi,

Could you please let us know what do you mean by not yet deployed? Does you mean that the jobs that has not been kicked off after you running the spark submit command (or) Could you please explain in detail.

Thanks

Akr

[#7]

If you are running with deploy mode = yarn (previously, master set to "yarn-client" or "yarn-cluster"), then you can discover the state of the spark job by bringing up the Yarn ResourceManager UI. In Ambari, select Yarn service from left-hand panel, choose "Quick Links", and click on "ResourceManager UI". It will open web page on port 8088.

Here is an example (click on 'Applications' in left panel to see all states):

[#8]

@Paul Hargis

Thanks for the info, appreciate your help on this.