Support Questions

frankmarit · ‎10-13-2016

What are people doing as far as version control and automated deployment for Nifi workflows?

What I'd like is to develop or modify the workflow in a dev environment, check it in to git, then import this workflow into the QA environment. After testing is complete, deploy this to the production nifi instance(s). It would be best if this process:

1) is able to be automated

2) would replace the existing workflow without needing to manual intervention; i.e. re-enter sensitive values like I have to do when importing templates.

thoughts?

alopresto · ‎10-13-2016

Hi Frank,

The development/QA/production environment promotion process (sometimes referred to as "SDLC" or "D2P" in conversation) is a topic of much discussion amongst the HDF development team. Currently, there are plans to improve this process in a future release. For now, I will discuss some common behaviors/workflows that we have seen.

The $NIFI_HOME/conf/flow.xml.gz file contains the entire flow serialized to XML. This file contains all processor configuration values, even sensitive values (encrypted). With the new Variable Registry effort, you can refer to environment-specific variables transparently, and promote the same flow between environments without having to update specific values in the flow itself.
The XML flow definition or specific templates can be committed and versioned using Git (or any other source code control tool). Recent improvements like "deterministic template diffs" have made this versioning easier.
The NiFi REST API can be used to "automate" the deployment of a template or flow to an instance of NiFi.
A script (Groovy, Python, etc.) could be used to integrate with both your source code control tool and your various NiFi instances to semi-automate this process (i.e. tap into Git hooks detecting a commit, and promote automatically to the next environment), but you probably want some human interaction to remain for verification at each state.

We understand that the current state of NiFi is not ideal for the promotion of the flow between dev/QA/prod environments. There are ongoing efforts to improve this, but I can't describe anything concrete at this time. If these points raise specific questions or you think of something else, please follow up.

View solution in original post

mail2msiva · ‎08-12-2019

Nifi_AutoDeploymentScript/ is really helpful in workflow deployment. However looking for more details on

1. controller services

2. reading variables of source process group and deploy only those variables per environment

3. reading json attributes

Cloudera Community

Support Questions

Nifi workflow version control & deployment