question Re: Oozie - Scheduling - Pig and Hive Scripts and Spark in Archives of Support Questions (Read Only)

Oozie - Scheduling - Pig and Hive Scripts and Spark

prodgers125 — Wed, 30 Nov 2016 19:05:31 GMT

Hi all,

Is possible to create an workflow on Oozie that automatically execute some Hive, Pig and Spark scripts in order to automate my analytics process?

Many thanks!

Re: Oozie - Scheduling - Pig and Hive Scripts and Spark

KuldeepK — Thu, 01 Dec 2016 03:37:17 GMT

@João Souza

Yes! You can use Hive/Pig/Spark actions in appropriate order as per your requirement and control the flow ( like if Hive is successful then more to Pig node and so on or else go to fail node. )

You can create a shell script and put calls to your Hive/Pig/Spark scripts in an appropriate order and use Oozie's shell action to execute the script.

If your cluster is Kerberized then I would not suggest to use shell action as it will create lot of issues related to authentication.

Hope this information helps! Happy Hadooping! 🙂

Re: Oozie - Scheduling - Pig and Hive Scripts and Spark

bandalora — Fri, 02 Dec 2016 04:54:05 GMT

You might want to look at Workflow Designer too, which is Technical Preview in HDP 2.5. You can work with it in the sandbox (http://hortonworks.com/downloads/#sandbox) and get an idea of how you can create Oozie workflows with Pig, Hive, and Spark actions.