question Re: List all created spark jobs in Archives of Support Questions (Read Only)

List all created spark jobs

bruce_a_perez — Mon, 23 May 2016 12:21:59 GMT

Hi,

I wanted to list all the spark jobs created. I tried looking in Resource Manager and Spark Job History Server and only active/failed/killed jobs are present. Is there a way to list all the spark jobs either running or not running? Could be in spark shell or any other that you can suggest.

Thanks.

Re: List all created spark jobs

sball — Mon, 23 May 2016 18:58:18 GMT

The Spark History server will have a list of all Jobs that have run using the YARN master.

If you are looking for current running jobs, the RM will give you a full list, though this will of course also include non-spark jobs running on your cluster.

If you are running spark standalone, you will not have any means of listing jobs.

Re: List all created spark jobs

phargis — Sun, 18 Aug 2019 11:48:40 GMT

The Spark History Server UI has a link at the bottom called "Show Incomplete Applications". Click on this link and it will show you the running jobs, like zeppelin (see image).

Re: List all created spark jobs

bruce_a_perez — Tue, 24 May 2016 09:58:13 GMT

Thanks @Simon Elliston Ball. But is there a way to see created spark jobs that has not been deployed yet?

Re: List all created spark jobs

bruce_a_perez — Tue, 24 May 2016 09:59:39 GMT

Thanks @Paul Hargis. Yes, I tried it also but what I wanted to see is the created spark jobs that has not yet been deployed. Is there a way to find this jobs?

Re: List all created spark jobs

phargis — Sun, 18 Aug 2019 11:48:32 GMT

If you are running with deploy mode = yarn (previously, master set to "yarn-client" or "yarn-cluster"), then you can discover the state of the spark job by bringing up the Yarn ResourceManager UI. In Ambari, select Yarn service from left-hand panel, choose "Quick Links", and click on "ResourceManager UI". It will open web page on port 8088.

Here is an example (click on 'Applications' in left panel to see all states):

Re: List all created spark jobs

bruce_a_perez — Thu, 26 May 2016 20:43:23 GMT

@Paul Hargis

Thanks for the info, appreciate your help on this.

Re: List all created spark jobs

AKR — Tue, 12 Nov 2019 14:16:55 GMT

Hi,

Could you please let us know what do you mean by not yet deployed? Does you mean that the jobs that has not been kicked off after you running the spark submit command (or) Could you please explain in detail.

Thanks

Akr