How to do NiFi Git Integration.?

[#1]

Hi,

I read there is development underway for NiFi Versioning , Builds & Deployment process. any timelines on this.?

In the meantime we are trying to see if we can integrate NiFi with Git for automated builds & deployments.

how to do that.? Can we check in\out at deployment object level (ex: template).? or Do we have to work at flow.xml.gz file level.??

Regards,Sai

[#2]

Hi Experts,

any help on how to integrate NiFi flows\template with Git for build and deployment purposes.?

Regards,Sai

[#3]

Hi @Saikrishna Tarapareddy

Indeed the community is working on a flow registry that should be available in coming releases of HDF/NiFi. I don't have a date for this yet, but I can see it available in the one of the next two releases.

For SDLC, you can integrate Git with flow.xml or with templates. IMO, using template is easier since you need to manage only a part of the flow. Since there are several flows running in NiFi, it's easier to deploy a template than the complete flow. This reduce impact on the production.

You can leverage NiFi API to implement your automated deployments.

Note that there are few technical challenges and it's not a lift/shift operation of templates. For instance, all passwords are protected when you download a template. You need to replacer them with the right value before deploying on the next environment. Another example is variables. Let's say you NiFi write data in Kafka, you need to change the Kafka broker address from env to another one. You can use custom variables with nifi.variable.registry.properties but this requires NiFi restart which is not acceptable for production. There's also a work on this topic : https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry

I hope these details help you.

Thanks

[#4]

Hi @Abdelkrim Hadjidj

Would you mind expanding on exactly how to use templates for flow control? Is it a case of creating a template, downloading that using the UI into a repo folder?

I also tried tracking and reverting changes on the flow.xml.gz but it didnt seem to revert the changes on the UI when I refreshed it?

Just wondering if im missing something here.

Thanks,

Tim

[#5]

Apache NiFi 1.5.0 + Apache NiFi Registry 0.1.0 offers a new approach to version control of flows, please take a look at these links:

https://bryanbende.com/development/2018/01/19/apache-nifi-how-do-i-deploy-my-flow

https://medium.com/@abdelkrim.hadjidj/fdlc-towards-flow-development-life-cycle-with-nifi-registries-...

[#6]

Hi @Abdelkrim Hadjidj

Thanks for your response on automating the deployments. I observed that after deployment, the environment variables still need to be updated manually. Is there a way to maintain environment specific variables in the registry and automate the deployments completely without manual intervention.

,

Hi @Abdelkrim Hadjidj

Thanks for your response on production deployments. I observed that in the blog after deployment to production, we have to manually update the properties. Is there a way to maintain environment specific variables in the registry and completely automate the deployments.

[#7]

Hi @Abhinav Yepuri

There are several ways to automate this. One of these is using NiFi CLI available from NiFi 1.6 : https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli

You have nifi pg-get-vars and nifi pg-set-var that you can use to get variables from dev, replace values with a dictionary and set in prod.