Referencing and reusing a single instance of a NiFi process group from multiple dataflows

[#1]

Hi All,

Is there a reusable process group in NiFi ? When I say reusable, I mean similar to how a function works, you define it once and call/use it many times, but it is defined in only 1 place; and if you change the function's code, the change is reflected in all places where the function is used; When I try to reuse the same process group in multiple dataflows, I'm actually getting separate copies/instances of the process group; if I make a change in the reusable process group, I want the change to be reflected in all dataflows where this reusable process group is used; however, what I'm finding in NiFi is, you've to make the change in each instance of the process group.

How do I get function like behavior when I want to reuse a process group ?

[#2 · Accepted solution]

Hi @Raj B,

At the moment, it is not possible to achieve what you are looking for. There is a feature proposal for that here:

https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups

You may also find the following of interest:

http://apache-nifi-users-list.2361937.n4.nabble.com/Process-Group-singleton-td535.html

Hope this helps.

[#3]

thanks @Pierre Villard; it's good to know it's in the works.

[#4]

you can use Input port and Output port for this purpose.. Data flow 1 will send data to PG.. Data flow 2 will also send data to the same PG.. this can be done using the Input Port within the PG and if there is any Output from the PG, add Output port as well within the PG.. The number of data flows connecting to this PG = the number of Input ports this PG has..

[#5]

Hi

Kylo is providing the functionality you are looking for

They have a concept together with nifi which is called reusable flow

This flow can be connected to any feed templates and in the end all feeds hook into the same reusable flow in nifi

So in short you create a flow in nifi with input port, create a template of this flow the reusable flow, export it and inport the file as a reusable flow in kylo.

Create your feed template flows in nifi and create template of them, import the template from nifi in kylo and connect the output port of your template to the input port of the reusable flow.

Create a feed in kylo from your template.

Kilo will then generate a feed based on your feed template in nifi and connect this to the reusable flow.

You can now modify your reusable flow and create a new template in nifi with the same name as the previous version of the reusable flow, export it to a file and import it to kylo and all your feeds are now connected to the new version of your reusable flow.

If you do not want to use Kylo there is an alternative

you create a group called reusable flow

In this group you create a group called reusable flow v1

You connect your feeds to the input port on the reusable flow group

And inside of the reusable flow you connect it to the v1 groups input port

Now you want to make a new version of your reusable flow

You make a copy of the group v1 to v2

Make the changes you want to do inside of v2, when you are satisfied you disconnect v1 and connect v2 instead.

In this way it is very easy to switch to a new version without having to change any of your feeds.

Below is link to youtube that describe the kylo reusable flow, but if you are only in nifi I recommend the groups in groups to maintain different versions as you then can easily switch back and forward without impacting all your feeds

https://www.youtube.com/watch?v=Vj641MRJCd8