NiFi integration with git

Question

I am trying to understand how to integrate NiFi within an existing SDLC. The following are some challenges that I am trying to resolve:

1) I am aware that NiFi Registry 0.2.0 allows us to add a git flow persistence provider. This means on every commit to the Registry, the flow file will be automatically be saved to the git repo. Is it possible to take the NiFi Registry out of the picture entirely and work solely with git. I have the feeling that there is a good reason to tightly integrate NiFi with NiFi registry, I would be really helpful if some one can explain the rational behind this tight integration.

2) Is it possible to deploy a NiFi flow from an artifactory instead of the NiFi Registry? A reference to an example would be really helpful.

3) Lets say I have a bunch of NiFi custom processors that needs to be hosted on git and there is Jenkins in this SDLC. Jenkins does the auto build of each custom processors and deploys them in the artifactory. Also the flow file, which would use these custom processors needs to be promoted to higher environments (dev, qa, prod) along with the Custom Processors (nar files). How should one design the SDLC for NiFi Custom processors and the flow file. Should all the custom processors and the flow files be in the same git repo? can each of the custom processors be part of the same project but be placed in their own package such that separate nar files are created while the flow file is also part of this project? Not sure I am making any sense any more at this point, please let me know if that is the case.

MattWho · Accepted Answer

@big-f

NiFi-Registry storage of version controlled flows is only the beginning use case for NiFi-registry.

1. Additional features of NiFi-Registry in the pipe line include:
- Storing of FlowFile parameters (NiFi parameters can be used by any property in any component in NiFi.)

- Storing of NiFi nars (NiFi has grown to be a very large application. Rather then installing every single component that exists, users can set up local registries that store nars. Then these nars can be pulled and dynamically loaded during runtime in NiFi as needed by the flow which has been built.)

2. The only method of deploying flows other than through NiFi-Registry is the legacy templates. but even here you have an existing NiFi canvas (blank or with other existing flows). You then import and instantiate templates on to that existing canvas somewhere, Downsides to this legacy method include no version control and templates are stored in heap memory. Even if a template is not instantiated to the canvas, it is held in heap memory. More templates equals more heap usage.

3. I think this question is answered by point number two in answer 1.

A NiFi FlowFile is what moves from processor to processor via a connection.
Not sure what you are referring to when you say "flow file"? Are you talking about the flow.xml.gz?

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.