Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

List all created spark jobs

Solved Go to solution

List all created spark jobs

New Contributor

Hi,

I wanted to list all the spark jobs created. I tried looking in Resource Manager and Spark Job History Server and only active/failed/killed jobs are present. Is there a way to list all the spark jobs either running or not running? Could be in spark shell or any other that you can suggest.

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: List all created spark jobs

The Spark History Server UI has a link at the bottom called "Show Incomplete Applications". Click on this link and it will show you the running jobs, like zeppelin (see image).

4453-sparkhistoryserverui-incompleteapps.png

7 REPLIES 7

Re: List all created spark jobs

Guru

The Spark History server will have a list of all Jobs that have run using the YARN master.

If you are looking for current running jobs, the RM will give you a full list, though this will of course also include non-spark jobs running on your cluster.

If you are running spark standalone, you will not have any means of listing jobs.

Re: List all created spark jobs

New Contributor

Thanks @Simon Elliston Ball. But is there a way to see created spark jobs that has not been deployed yet?

Re: List all created spark jobs

The Spark History Server UI has a link at the bottom called "Show Incomplete Applications". Click on this link and it will show you the running jobs, like zeppelin (see image).

4453-sparkhistoryserverui-incompleteapps.png

Re: List all created spark jobs

New Contributor

Thanks @Paul Hargis. Yes, I tried it also but what I wanted to see is the created spark jobs that has not yet been deployed. Is there a way to find this jobs?

Re: List all created spark jobs

Cloudera Employee

Hi,

 

Could you please let us know what do you mean by not yet deployed? Does you mean that the jobs that has not been kicked off after you running the spark submit command (or) Could you please explain in detail.

 

 

Thanks

Akr

Re: List all created spark jobs

If you are running with deploy mode = yarn (previously, master set to "yarn-client" or "yarn-cluster"), then you can discover the state of the spark job by bringing up the Yarn ResourceManager UI. In Ambari, select Yarn service from left-hand panel, choose "Quick Links", and click on "ResourceManager UI". It will open web page on port 8088.

Here is an example (click on 'Applications' in left panel to see all states):

4494-yarn-resourcemanager.png

Re: List all created spark jobs

New Contributor
@Paul Hargis

Thanks for the info, appreciate your help on this.

Don't have an account?
Coming from Hortonworks? Activate your account here