Hortonworks HDF - Apache Nifi - how to deploy a flow from dev environment to production environment

[#1]

Dear All,

I just started Hortonworks HDF - Apache Nifi - developed first Flow. Now wanted to deploy this code from dev to production or Qa environment.

I read documentation and other articles I can select a sub-section of a flow, create a template and import it.

But my requirement is to deploy the whole flow I developed from one environment (dev) to another environment (say QA server).

Also how can I change the proper values at runtime based on the environment (dev vs QA vs PROD) without changing the flow logic.

I read flow.gz file but I did not understand basically how can I manage multiple flows within my web page.

Basically how can I create an installer similar to .war or .ear or .targ.gz which contains all the artifacts for a particular data flow (say ingesting change data capture from RDBMS into HDFS). When deployed and executed in different environments (say dev or qa or prod) the same code is deployed but different configuration parameters are provided at runtime.

Reason for this is in a large enterprise once a code is tagged in SVN and installer is prepared for deployment, IT support team will not provide access to QA or PROD cluster to developers. And everything is automated through Hudson / DevOps tools where by click of a button code is deployed on higher environments. We cannot expect IT support team to import templates, drag and drop to develop the Flows on prod environment. They expecting everything is developed with automation at the backdrop (automation I mean Hudson/DevOps tool click of adeployment, changing runtime configuration values with an external configuration file etc)

Please help me, above is very important to share my views on Nifi to large group within my dev team.

Thanks,

[#2 · Accepted solution]

This article will show you how to set variables per environment, including sensitive values so you can promote a single change-managed template of your full flow from one environment to another without changing contents of the template. Also shows how to build reusable components. (The SDLC model is at the bottom of the article).

https://community.hortonworks.com/articles/60868/enterprise-nifi-implementing-reusable-components-a....

Process is:

1. In flow configurations, sensitive values should be configured as Expression Language references to OS environment variables that you set in each environment, e.g. ${MYSQL_PSSWRD}.

   Other environment-specific config values should similarly use Expression Language references. If these are not sensitive, should be in custom properties file.

2. Developers finalize their flows and submit the template of the flow to version control, eg Git (and also submit custom property files).
3. Template and custom property files are promoted to each environment just as source code typically is.
4. Automation: deploying templates to environments can be done via NiFi RestAPI integrated with other automation tools.
5. Governance bodies decide which configurations can be changed in real-time (e.g. ControlRate properties). These changes do not need to go through verision control and can be made by authorized admins on the fly.

The article elaborates on Expression language, OS environment variables and custom property files. If you have specific questions stemming from the article, please continue as comments in this thread.

Note: To deal with all flows as one deployable unit, you can make them all a part of a single processing group and change-manage that as a single template. Otherwise you can manage each flow as separate templates. Or you can do something in between: form process groups as logical groups (e.g. team, project, product, line of business) and use these as deployable units.

[#3]

@Greg Keys, if I update a property in my custom properties file, will nifi automatically pick up the changes or do I have to restart the nifi service or the processors using that property?

[#4]

Hi @Jatin Kheradiya, I suggest placing this as a standalone question to get full visibility and value for the committee. 🙂

[#5]

You can go see this application : https://github.com/hermannpencole/nifi-config

It can update and extract a process group Configuration (with childrens)

It will be done without restarting nifi, it use nifi webservice.