Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

how to trigger sample spark and pyspark jobs using oozie?

SOLVED Go to solution

how to trigger sample spark and pyspark jobs using oozie?

Explorer
 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: how to trigger sample spark and pyspark jobs using oozie?

Contributor

You can use the Spark Action in Oozie to submit any spark applications:

https://archive.cloudera.com/cdh5/cdh/5/oozie/DG_SparkActionExtension.html#Spark_Action

 

If you are more familar with spark-submit tool,  you can try to use oozie shell action as well:

https://archive.cloudera.com/cdh5/cdh/5/oozie/DG_ShellActionExtension.html

 

You may need to make sure the spark gateway role is deployed on the oozie server and node manager nodes, so that the runtime env always have the depencies available.

1 REPLY 1

Re: how to trigger sample spark and pyspark jobs using oozie?

Contributor

You can use the Spark Action in Oozie to submit any spark applications:

https://archive.cloudera.com/cdh5/cdh/5/oozie/DG_SparkActionExtension.html#Spark_Action

 

If you are more familar with spark-submit tool,  you can try to use oozie shell action as well:

https://archive.cloudera.com/cdh5/cdh/5/oozie/DG_ShellActionExtension.html

 

You may need to make sure the spark gateway role is deployed on the oozie server and node manager nodes, so that the runtime env always have the depencies available.