Changing NiFi Templates

[#1]

Hi,

i have a couple of NiFi templates for Logging and emails which i use in flows many times based on conditions (like Errors,Status etc).if i have to make a change to template , i need to open all the places that the template is being used to change those too.

is that correct.? i was hoping to change it at one place (in main template of something) and it would change all the instances of the template.

[#2]

@Saikrishna Tarapareddy

A template is nothing more then a collection of NiFi processors, connections and controller services that includes all their unique configurations (except sensitive properties). There is no direct correlation between a template and any processors on the canvas.

When you use a template to instantiate the components contained within it to the canvas, there is no back reference to the template. You are free to modify, delete, etc any component instantiated from that template. In addition, each component when instantiated gets its own unique UUID assigned to it.

A good analogy would be a Word document. If you create a Word document named "mydoc1.docx" and then later decide to select "Save as." and save a copy as "mydoc2.docx". Going back to mydoc1 and making changes will not apply to mydoc2. NiFi templates behave similarly.

Once NiFi has successfully instantiated a template to the canvas it never goes back and looks at that template again to do any sort of comparison.

It would be dangerous for NiFi to assume what changes should and should not be applied if it did such a comparison. Would you want it to alway blank sensitive properties. It is very common that adjustments are made to components instantiated form a templaet post being created. So the same template instantiated multiple times within the same NiFi is very likely to diverge in configuration from each other over time.

Thanks,

Matt

[#3]

@Saikrishna Tarapareddy

I understand you concern. There is no notion of a template when looking at the components loaded on your canvas. A Template simply allows you to instantiate a collection of components in preconfigured setup to the canvas. Once those components are instantiated, they are treated no different then any other component you added manually. The underlying process is no different.

While this is a valid use case, simply changing how this process works could have serious issues for those who do not expect it to work this way. This use case requires an entirely new process.

There is work ongoing in the NiFi community to develop versioned templates (right now there is no notion of a version on templates you create). This new feature would allow you to create version controlled variations of a template. As part of this the instantiated components would be tied back to a specific template version. I do not know all the details of where this is going or when it will become reality, but should make it easier to swap out versions of the same template instantiated to your canvas.

Thank you,

Matt

[#4 · Accepted solution]

@Matt Clarke

Thanks for the quick reply , while I understand the consequences of changing template instances when main template is changed.

i think there will be some cases where we want this functionality , for example in my case i am logging jobname,filename,date,status,comments to a file from all my flows. i created a template where it accepts all these as attributes\variables and using it.

But if i want to add fileSize attribute also to the logevent, i should be able to open the template flow and and add it without touching each instances as fileSize is a default attribute.

so i think we need two types of templates , first type where we can change and second type where we cannot change the instantiated template. for the second type we can change at one place and propagate the changes to where ever it is used.

Hope this makes sense..

Regards,

Sai

[#5]

@Saikrishna Tarapareddy, you may want to create a process group that contains the instantiation of your template, and then create connections from the areas of your flow to that process group. That way, you have one "instance" of the template created, and you'll only need to do your modifications once. You can always save a new template (for other instantiations, exporting, etc) with your modifications. I do admit that making connections across process groups in order to reuse a specific group may make the flow a bit harder to read, but eventually NiFi will support some improvements to make this easier/cleaner to do in a flow.

[#6]

I must say, I'm pretty shocked that this is how templates work. As a new user, I expected this from templates:

• Ability to open/edit a template (no option from the Templates list).
• Have a template hold its reference when used. Consider an "authenticate to X service" template which could be used in many Process Groups, but given different credentials for each.

Maybe I'm thinking of templates wrong and what I really needs is an "authenticate to X service" Process Group and have my other Process Groups use it?

[#7]

@J_123 
This is a 6 year old post.  NiFi Templates are close to point of deprecation from NiFi.  A NiFi is and has always been a snapshot of a specific set of processors.  It has no mechanism for tracking its reuse within the same NiFi where it was used.  Lets say someone creates a template, reuses it on the same NiFi multiple times, and then deletes the template from the NiFi (which is recommended since template like the rest of the flow on he canvas reside in NIFi's heap memory), how then do we track and replicate those changes.  Memory usage and this tracking are a few reasons why templates are in the way out and why we no longer recommend using this capability.

Back at the end of 2021 (long after this post), NiFi introduced a new NiFi-Registry[1] service that is installed separate form NiFi and integrates with the NiFi service.  NiFi-Registry allows users to build dataflows on the a NiFi canvas and then version control each of those flows into NiFi-Registry.  Once version controlled into NiFi-Registry, that version controlled flow can be reuses on the same NiFi or any other NiFi using that same NiFi-Registry. If a user makes a change to anyone of these version flows linked to version control, NiFi will prompt that local changes exist allowing user to commit those locL changes to the copy in the NiFi-Registry.  Then all the other places (across multiple NiFi's if using that same dataflow from NiFi-Registry) where that version controlled flow is being used will prompt that a newer version is available allowing users to apply those changes.

If you have questions about something in NiFi and you cannot find a recent thread on the topic to raise a new question in the community to get the most up-to-date information.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[1] https://nifi.apache.org/docs/nifi-registry-docs/index.html