I currently use spark submit to execute an SBT created jar.
How can I automate the process in order to let it start periodically (i.e. every night or every few hours?)
Hi Ir, one way is to use Oozie to schedule a Spark job. It supports launching in a Spark Action. In Ambari you can setup the Workflow Manager view.
Here are some docs on setting it up: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_workflow-management/content/ch_wfm_basic...
Let us know how it goes!