How can i use an already existing controller service when I deploy my template to Nifi

[#1]

Currently, every time I deploy a template from one environment to another (or edit something and upload to the same environment) , it deploys the controller services (even if they are already existing with the same name) onto the environment at the Process Group level. Would it be possible to reuse the existing controller services (DBCP connection pool for example) rather than creating a new one at the process group level everytime ?

I have tried editing the xml and replaced the controller Node with pre-used Controller Node, but this does not help. Right now, as the Nifi Framework is growing, this task is becoming cumbersome and difficult to manage.

[#2 · Accepted solution]

@Tanmoy

NiFi templates where never intended when they were designed for the purpose of version control. A template by design bundles everything that the selected components need to work. Also to make sure templaet could be deployed to any other environment, all sensitive properties are cleared.

The NIFi team realized that design made it less then ideal for dataflow promotion between environments.

That is where the NiFi Registry came int to the picture (Available as of Apache NiFi 1.5 and HDF 3.1.x)

https://nifi.apache.org/registry.html

The NIFi Registry runs as an external service but complimentary service to NiFi. Multiple NiFi instances/clusters can be configured to access the same registry. User can start version control on process groups within NiFi which stores a version controlled copy of the flow (similar to a template) in the NIFi Registry.

User on another NiFi instance/cluster can connect to the same registry and load that flow on to their NiFi. Initially adding a flow form the registry works just like templates (Need to enter sensitive properties and controller services are copied over.) What is different is how that info is preserved through version upgrades from that point forward.

Let say on system one (dev) and new version of that version controlled process group is pushed to the registry. On System two they can upgrade to version 2 without needing to reset sensitive properties or worry about getting another copy of a controller service.

Thank you,

Matt

*** If you found this answer addressed your initial question, please take a moment to login and click on "Accept".

[#3]

Thanks @Matt Burgess for your reply and for sharing the link about registry. I would look into this. For the time being, we dont plan to do a upgrade of Nifi. Any work around that you would suggest in the interim ?

EDIT : Shout out for @Matt Clarke

[#4]

Thank you, but I am not @Matt Burgess LOL

Forum TIP: Try to avoid responding to answers with new answers. Instead add comments to answers you are responding to.

[#5]

@Tanmoy

There is no simple workaround. You would need to create some scripts to manually update your flow after it is instantiated to point the newly added processors to an existing Controller services and delete the unneeded controller services.

-

you may find this link helpful in pointing you in the right direction:

https://community.hortonworks.com/repos/22817/nifi-deployment-automation.html

-

Thanks,

Matt

[#6]

moved to comment Matt. Sorry about the confusion, just realized it 😛