Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

[#1]

Hello all;

I have separate sqoop workflow, hive workflow and spark workflow. Is there anyway I can combine these three workflows into one Oozie job? if so, What should I use? Oozie Coordinator or Bundle? Please advise

Thanks

[#2 · Accepted solution]

@JT Ng

Unfortunately, one oozie coordinator can only have one action/workflow. Therefore, you would create a Coordinator for each of your workflows (Sqoop, Hive, Spark, etc...), and then link them with a bundle.

A Bundle is a higher-level oozie abstraction that will batch a set of coordinator applications, creating a data pipeline. In bundle.xml you can set multiple coordinator jobs by which multiple workflows can be launched.

If you're using Ambari 2.5+, you can use Workflow Manager to do this. Take a look at the link below:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_workflow-management/content/sect_create_...

(Workflow Manager is also available as tech preview in Ambari 2.4.x)

[#3]

You answered my question. Many Thanks

[#4]

Hi Eyad,

I have Ambari Version 2.4.0.1 and HDP Version 2.5.3.0-37. Is there anyway I can upgrade Ambari to 2.5+ without upgrading whole HDP? Please advise.

Thanks

[#5]

@JT Ng

Yes, you can upgrade Ambari independently of HDP. The latest version of Ambari is 2.5.1 so I recommend you move to that. Below is a link to the documentation where you will find more information about it as well as upgrade instructions (just skip the HDP upgrade parts).

https://docs.hortonworks.com/HDPDocuments/Ambari/Ambari-2.5.1.0/index.html

[#6]

One single workflow can contain several actions. But you need to make sure all dependencies are prepared for those actions