question Re: How to use NiFi to launch a Oozie workflow job? in Support Questions

How to use NiFi to launch a Oozie workflow job?

dsun — Sun, 26 Feb 2017 00:49:53 GMT

Please help me understand the best practices around launching a Oozie workflow job by using NiFi. For instance, once I write data into HDFS by using a PutHDFS processor, I assume I should use an ExecuteScript processor to explicitly invoke a Oozie workflow job, either via Java client API/REST API, or Oozie CLI depending if NiFi instance resides inside the HDP cluster or not.

Thanks,

Derek

Re: How to use NiFi to launch a Oozie workflow job?

aervits — Sun, 26 Feb 2017 01:04:28 GMT

though you can achieve what you're planning it really goes against what both these products are designed to do. Oozie uses coordinator and bundle to schedule their workflows, Nifi is about always flowing data without start and finish. Oozie does have rest API you can invoke to start a workflow if you intend to go your route but I would first ask what user is trying to do. If you need a way to check whether data landed in HDFS before executing Oozie workflow, look at the following coordinator examples by Yahoo, specifically https://github.com/yahoo/oozie/wiki/Oozie-Coord-Use-Cases#triggering-coordinator-jobs-when-data-directory-is-available

Also suggest you read my articles https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-for-apache-ooz-5.html and

https://community.hortonworks.com/articles/85361/apache-ambari-workflow-manager-view-for-apache-ooz-6.html

Re: How to use NiFi to launch a Oozie workflow job?

dsun — Sun, 26 Feb 2017 01:12:49 GMT

I was just trying to understand if there is any native NiFi & Oozie integration out of the box, such as NiFi & Flume, guess not. Thanks Artem.

Re: How to use NiFi to launch a Oozie workflow job?

aervits — Sun, 26 Feb 2017 01:15:36 GMT

You can certainly achieve that, or contribute a patch but it defeats the purpose. If you need more explanation, ping me on hipchat. Don't forget to accept the answer!

Re: How to use NiFi to launch a Oozie workflow job?

dsun — Sun, 26 Feb 2017 02:00:30 GMT

Will chat more on hipchat if needed. Thanks.

Re: How to use NiFi to launch a Oozie workflow job?

adelgacem — Wed, 07 Mar 2018 18:02:38 GMT

Hello,

I got the same remark/question/proposal that i would expose differently :

==> OOZIE may suffer after a certain number of workflows, if the triger is Data availabilty based.

The Question (or answer) is :

When using NIFI to ingest data, and instead of puting a "SUCCESS" flag to trigger OOZIE WFL, why not lauch the WFL trought a NIFI InvokHTTP Processor simply ?

( and routing failures to a 2nd InvokHttp or loop to the same to relaunch during a period (expire attribute link) in case of network pb for example).

I tested is and it's wotks fine. The question is there any against indications ?

Best.

Re: How to use NiFi to launch a Oozie workflow job?

chhaya_vishwaka — Tue, 03 Sep 2019 01:50:18 GMT

@adelgacem can you share sample for how you invoked Oozie using InvokHttp processor? I'm trying to run Oozie workflow from NIFI it will be a great help if you can share NIFI processor configuration details.