Support Questions
Find answers, ask questions, and share your expertise

How can I execute Spark jobs periodically?

How can I execute Spark jobs periodically?

Explorer

Hello,

I currently use spark submit to execute an SBT created jar.

How can I automate the process in order to let it start periodically (i.e. every night or every few hours?)

Thank you

3 REPLIES 3
Highlighted

Re: How can I execute Spark jobs periodically?

Explorer

Hi Ir, one way is to use Oozie to schedule a Spark job. It supports launching in a Spark Action. In Ambari you can setup the Workflow Manager view.

Here are some docs on setting it up: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_workflow-management/content/ch_wfm_basic...

Let us know how it goes!

Highlighted

Re: How can I execute Spark jobs periodically?

Explorer
Highlighted

Re: How can I execute Spark jobs periodically?

Explorer

thank you Mark