question Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ? in Archives of Support Questions (Read Only)

Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

jeanette75007 — Tue, 06 Jun 2017 02:16:58 GMT

Hello all;

I have separate sqoop workflow, hive workflow and spark workflow. Is there anyway I can combine these three workflows into one Oozie job? if so, What should I use? Oozie Coordinator or Bundle? Please advise

Thanks

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

egarelnabi — Tue, 06 Jun 2017 02:42:20 GMT

@JT Ng

Unfortunately, one oozie coordinator can only have one action/workflow. Therefore, you would create a Coordinator for each of your workflows (Sqoop, Hive, Spark, etc...), and then link them with a bundle.

A Bundle is a higher-level oozie abstraction that will batch a set of coordinator applications, creating a data pipeline. In bundle.xml you can set multiple coordinator jobs by which multiple workflows can be launched.

If you're using Ambari 2.5+, you can use Workflow Manager to do this. Take a look at the link below:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_workflow-management/content/sect_create_coord_bundle.html

(Workflow Manager is also available as tech preview in Ambari 2.4.x)

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

jeanette75007 — Tue, 06 Jun 2017 07:43:59 GMT

You answered my question. Many Thanks

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

jeanette75007 — Tue, 06 Jun 2017 20:40:19 GMT

Hi Eyad,

I have Ambari Version 2.4.0.1 and HDP Version 2.5.3.0-37. Is there anyway I can upgrade Ambari to 2.5+ without upgrading whole HDP? Please advise.

Thanks

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

egarelnabi — Tue, 06 Jun 2017 20:46:46 GMT

@JT Ng

Yes, you can upgrade Ambari independently of HDP. The latest version of Ambari is 2.5.1 so I recommend you move to that. Below is a link to the documentation where you will find more information about it as well as upgrade instructions (just skip the HDP upgrade parts).

https://docs.hortonworks.com/HDPDocuments/Ambari/Ambari-2.5.1.0/index.html

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

xin_wang — Tue, 18 Jul 2017 00:27:34 GMT

One single workflow can contain several actions. But you need to make sure all dependencies are prepared for those actions