Nifi workflow version control & deployment

[#1]

What are people doing as far as version control and automated deployment for Nifi workflows?

What I'd like is to develop or modify the workflow in a dev environment, check it in to git, then import this workflow into the QA environment. After testing is complete, deploy this to the production nifi instance(s). It would be best if this process:

1) is able to be automated

2) would replace the existing workflow without needing to manual intervention; i.e. re-enter sensitive values like I have to do when importing templates.

thoughts?

[#2 · Accepted solution]

Hi Frank,

The development/QA/production environment promotion process (sometimes referred to as "SDLC" or "D2P" in conversation) is a topic of much discussion amongst the HDF development team. Currently, there are plans to improve this process in a future release. For now, I will discuss some common behaviors/workflows that we have seen.

• The $NIFI_HOME/conf/flow.xml.gz file contains the entire flow serialized to XML. This file contains all processor configuration values, even sensitive values (encrypted). With the new Variable Registry effort, you can refer to environment-specific variables transparently, and promote the same flow between environments without having to update specific values in the flow itself.
• The XML flow definition or specific templates can be committed and versioned using Git (or any other source code control tool). Recent improvements like "deterministic template diffs" have made this versioning easier.
• The NiFi REST API can be used to "automate" the deployment of a template or flow to an instance of NiFi.
• A script (Groovy, Python, etc.) could be used to integrate with both your source code control tool and your various NiFi instances to semi-automate this process (i.e. tap into Git hooks detecting a commit, and promote automatically to the next environment), but you probably want some human interaction to remain for verification at each state.

We understand that the current state of NiFi is not ideal for the promotion of the flow between dev/QA/prod environments. There are ongoing efforts to improve this, but I can't describe anything concrete at this time. If these points raise specific questions or you think of something else, please follow up.

[#3]

Thanks @Andy LoPresto. If I go the flow.xml.gz route, it looks like I have to shut down the currently running nifi instance, then copy the flow.xml.gz file to the conf directory and then restart the server. Is that right? When I tried to copy it over while the server was running it didn't appear to pick up the changes.

The variable registry page you linked didn't really have much content at all. Is that under development or still just being discussed?

[#4]

Frank,

• Yes, you are correct that replacing the flow definition requires restarting the server.
• The VR wiki page is a work in progress, as is the development effort. @Yolanda M. Davis has done significant work on this and more information is available in the Getting Started Guide and Admin Guide.

[#5]

NiFI Deploy

https://github.com/aperepel/nifi-api-deploy

you can have a git script check-in every XML you see in the conf/archive directory

or manually save templates when you have work done at the end of a day (and check-in those XML)

[#6]

@Frank Maritato

I have recently published an article on this topic: https://community.hortonworks.com/content/kbentry/60868/enterprise-nifi-implementing-reusable-compon...

This is meant to realize the SDLC in the style of conventional code change-management and deployment, and is best seen as an alternative to the post of @Andy LoPresto. It is meant to tie into an enterprise's existing process for code in general.

Main suggestions are:

• In configs of flows, sensitive values should be configured as Expression Language references to OS environment variables that you set in each envt. Ex. ${MYSQL_PSSWRD}
• Other environment specific config values should similarly use Expression Language refs. If these are not sensitive, should be in custom properties file.
• Developers finalize their flows and submit the template of the flow to git (and the custom property file). Template and custom property file is promoted to each environment just as source code typically is.
• Automation: uploading finalized (git) template to environments can be done via NiFi RestAPI

[#7]

@Greg Keys

How to do the third step in your flow (copied below)

• Developers finalize their flows and submit the template of the flow to git (and the custom property file). Template and custom property file is promoted to each environment just as source code typically is.

if we select the components we want to make as a template to promote to higher environments , is there a way to check-in\submit that flow(template) to git instead of the whole flow.xml.gz?

also any good article of integrating NiFi flows with git.??

Regards,

Sai

[#8]

@Saikrishna Tarapareddy Keep in mind that "flow" does not have to mean all components on the canvas. For example, a flow could be all components in a process group. In that case, you would deploy only the process group in question to deploy your flow. So the unit of deployment, what we are calling flow, is up to you. It could be one independent flow or many or all.

For better visibility to the community, I would pose your NiFi git integration question as a new question.

Please note that there is a lot of activity in the NiFi project around SDLC. Future plans are to have the repository embedded in the NiFi platform itself so you will not have to import and export templates, but can rather code manage them on NiFi.

[#9]

@Greg Keys

Thank you . my doubt was how do you submit template of flow to git automatically..??

Will post NiFi Git integration as a separate question.

Regards,Sai

[#10]

You can use Nifi-Toolkit which provides CLI to interact with NIFI instance and NIFI Registry instance.

A Powershell script can than be written to automate the whole deployment process.

You need to register registry clients to you Nifi instance. Once you create the appropriate buckets per your environment in your registry instance and connect this instance to your Nifi flows, you can use Nifi toolkit which provides commands to export flow from one environment and import it to other via registry buckets.

e.g. run the following command from the Cli.sh or cli.bat which is found in nifi-toolkit code base.

registry export-flow-version -f $fromFlowidentifier -fv $flowversion -o temp-flow.json -ot json -u $registrylink ;

here fromflowidentifier is the flow identifier that is found in your registry bucket. temp.json is the file where it will export the flow. Once exported you can import the flow using bellow command.

registry import-flow-version -f $toflowidentifier -i temp-flow.json -u $registrylink ;

Here it is assumed you have linked the nifi instances with the buckets in registryclient .

Here is my sample autodeployment powershell script to import nifi flow: It will not only deploy the flow but will also set the flow variables. If deployment fails for some reason it will also send an email. This assume you are using Nifi-registry and have installed Nifi-toolkit. Please go through the readme and script is also commented for proper understanding.

Here is the example powershell script: https://anp8729.github.io/Nifi_AutoDeploymentScript/