Support Questions
Find answers, ask questions, and share your expertise

Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

Solved Go to solution
Highlighted

Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

Hello all;

I have separate sqoop workflow, hive workflow and spark workflow. Is there anyway I can combine these three workflows into one Oozie job? if so, What should I use? Oozie Coordinator or Bundle? Please advise

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

@JT Ng

Unfortunately, one oozie coordinator can only have one action/workflow. Therefore, you would create a Coordinator for each of your workflows (Sqoop, Hive, Spark, etc...), and then link them with a bundle.

A Bundle is a higher-level oozie abstraction that will batch a set of coordinator applications, creating a data pipeline. In bundle.xml you can set multiple coordinator jobs by which multiple workflows can be launched.

If you're using Ambari 2.5+, you can use Workflow Manager to do this. Take a look at the link below:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_workflow-management/content/sect_create_...

(Workflow Manager is also available as tech preview in Ambari 2.4.x)

View solution in original post

5 REPLIES 5
Highlighted

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

@JT Ng

Unfortunately, one oozie coordinator can only have one action/workflow. Therefore, you would create a Coordinator for each of your workflows (Sqoop, Hive, Spark, etc...), and then link them with a bundle.

A Bundle is a higher-level oozie abstraction that will batch a set of coordinator applications, creating a data pipeline. In bundle.xml you can set multiple coordinator jobs by which multiple workflows can be launched.

If you're using Ambari 2.5+, you can use Workflow Manager to do this. Take a look at the link below:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_workflow-management/content/sect_create_...

(Workflow Manager is also available as tech preview in Ambari 2.4.x)

View solution in original post

Highlighted

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

You answered my question. Many Thanks

Highlighted

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

Hi Eyad,

I have Ambari Version 2.4.0.1 and HDP Version 2.5.3.0-37. Is there anyway I can upgrade Ambari to 2.5+ without upgrading whole HDP? Please advise.

Thanks

Highlighted

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

@JT Ng

Yes, you can upgrade Ambari independently of HDP. The latest version of Ambari is 2.5.1 so I recommend you move to that. Below is a link to the documentation where you will find more information about it as well as upgrade instructions (just skip the HDP upgrade parts).

https://docs.hortonworks.com/HDPDocuments/Ambari/Ambari-2.5.1.0/index.html

Highlighted

Re: Can I combine Sqoop workflow, Hive workflow and Spark workflow into one Oozie workflow job ?

Explorer

One single workflow can contain several actions. But you need to make sure all dependencies are prepared for those actions