Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi template use existing controller service when promoting code to different environments

NiFi template use existing controller service when promoting code to different environments

Contributor

I want to be able to promote code throughout our environments without having to manually update any of the processors in the UI after deployment. So far, I've seen people talk about copying the flow.xml.gz file over and restarting the service, but I don't see this working in a real world scenario because you would have to promote the entire environment and not just the code changes that were ready to be deployed. Another option, which I think is more likely, is to create a temple, export it, and import it into the next environment. This works well, and I can use the custom properties file with the expression language to handle most of the configuration changes, but when I import a template that references a controller service, it always creates a new controller service instead of using the existing one. This happens even when I copy and past a process group within the same environment or create a new process group from a template within the same environment. I end up having to delete that controller service that is created automatically and update the processor to use the existing controller service. What I would like to do is export the template from UAT, for example, and have a script that would use a mapping of controller service ids from UAT to the matching controller service in PROD and update that value in the XML, so when I import the template it finds the existing controller service instead of creating a new one, but when I update the controller service id and then import it, it updates the id again automatically and then creates a new controller service instead of using the existing. Has anyone dealt with this type of issue before and have any ideas of workarounds?

2 REPLIES 2

Re: NiFi template use existing controller service when promoting code to different environments

New Contributor

Yes, I have been hampered by the same issue with new controllers being created as you described. Promoting the flow.xml.gz between environments seems to have little real-world support in NiFi to make it easy to do. People say Expression Language (EL) is the answer, but then you look and many properties do not support Expression Language. If a single required property that varies between your environments does not support EL (and many passwords do not support it), then EL fails as a solution. Furthermore even if EL were implemented for every ui editable processor property, it is a major issue that the remote input/output ports embed your NiFi server names when they are created and can not be changed without being deleted and having a new port created in its place. So until major enhancements to NiFi are added to address these issues, promoting across environments will be a huge headache and it greatly weakens the enterprise readiness of NiFi. I consider this the largest issue hindering using NiFi in large systems with different environments.

Please enlighten me if I have missed something. I truly am impressed with NiFi and it does many great things, but this is an unfortunate flaw.

Highlighted

Re: NiFi template use existing controller service when promoting code to different environments

New Contributor

The issue has been around for a while, but perhaps voting on it might get some movement: https://issues.apache.org/jira/browse/NIFI-3177