Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can I execute Spark jobs periodically?

How can I execute Spark jobs periodically?

New Contributor

Hello,

I currently use spark submit to execute an SBT created jar.

How can I automate the process in order to let it start periodically (i.e. every night or every few hours?)

thank you

1 REPLY 1
Highlighted

Re: How can I execute Spark jobs periodically?

Rising Star

@Ir Mar Starting with HDP 2.6, you can use Workflow Designer to design and schedule work flows, including Spark jobs. Documentation is at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_workflow-management/content/ch_wfm_basic.... Alternatively, you can use Oozie to schedule Spark workflows, and details around that can be found at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_spark-component-guide/content/ch_oozie-s.... Hope this helps, and please remember to upvote / accept the answer if you found this useful.

Don't have an account?
Coming from Hortonworks? Activate your account here