NiFi integration with git

[#1]

I am trying to understand how to integrate NiFi within an existing SDLC. The following are some challenges that I am trying to resolve:

1) I am aware that NiFi Registry 0.2.0 allows us to add a git flow persistence provider. This means on every commit to the Registry, the flow file will be automatically be saved to the git repo. Is it possible to take the NiFi Registry out of the picture entirely and work solely with git. I have the feeling that there is a good reason to tightly integrate NiFi with NiFi registry, I would be really helpful if some one can explain the rational behind this tight integration.

2) Is it possible to deploy a NiFi flow from an artifactory instead of the NiFi Registry? A reference to an example would be really helpful.

3) Lets say I have a bunch of NiFi custom processors that needs to be hosted on git and there is Jenkins in this SDLC. Jenkins does the auto build of each custom processors and deploys them in the artifactory. Also the flow file, which would use these custom processors needs to be promoted to higher environments (dev, qa, prod) along with the Custom Processors (nar files). How should one design the SDLC for NiFi Custom processors and the flow file. Should all the custom processors and the flow files be in the same git repo? can each of the custom processors be part of the same project but be placed in their own package such that separate nar files are created while the flow file is also part of this project? Not sure I am making any sense any more at this point, please let me know if that is the case.

[#2 · Accepted solution]

@big-f 

NiFi-Registry storage of version controlled flows is only the beginning use case for NiFi-registry. 

1. Additional features of NiFi-Registry in the pipe line include:
- Storing of FlowFile parameters (NiFi parameters can be used by any property in any component in NiFi.)

- Storing of NiFi nars (NiFi has grown to be a very large application.  Rather then installing every single component that exists, users can set up local registries that store nars.  Then these nars can be pulled and dynamically loaded during runtime in NiFi as needed by the flow which has been built.)

2. The only method of deploying flows other than through NiFi-Registry is the legacy templates.  but even here you have an existing NiFi canvas (blank or with other existing flows). You then import and instantiate templates on to that existing canvas somewhere,  Downsides to this legacy method include no version control and templates are stored in heap memory. Even if a template is not instantiated to the canvas, it is held in heap memory.  More templates equals more heap usage.

3. I think this question is answered by point number two in answer 1.  

A NiFi FlowFile is what moves from processor to processor via a connection.
Not sure what you are referring to when you say "flow file"? Are you talking about the flow.xml.gz?

[#3]

Thank you so much for the answers, much appreciated.  

By flow files I mean what is being stored in the regsitry (a flow file that resides in a bucket referred by FLOW IDENTIFIER).

Would you be able to tell me approximately when the " Storing of NiFi nars" in the Registry will be available?

Thank you so much!

[#4]

@big-f 

The work to support bundles is part of NiFi-Registry 0.4.0

https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#bundle-persistence-pr...
https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html#manage-bundles

https://issues.apache.org/jira/browse/NIFIREG-211

[#5]

hello,

I see there is POST option to save a bundle/nar into the NiFi Registry however that can only happen from a local machine.   Is it possible to fetch bundle/nar from an artifactory and be able to save it in the NiFi Registry?  The reason I am thinking in that direction is because in our current SDLC process, Jenkins will do the build and resolve all dependencies and create the NAR and put it in the Artifactory.  How does the bundle gets deployed from the artifactory into the Registry, that is what I am after. 

Thanks!