I've created a datflow on the canvas, now how do I save it and deploy to production? -Newbie

[#1 · Accepted solution]

@Blake Colson

The answer to that question lies in what version of NiFi you are running....

I will assume you are running the latest version in this response. (NiFI 1.2.0 or HDF 2.1.2 as of time this was written).

You have two option for moving your entire dataflow from one system to another.

1. Copy the flow.xml.gz file from one NIFi instance to another.

- This method requires that both NiFi instances use the same configured sensitive props key in their nifi.properties files. The sensitive props key is used to encrypt the sensitive properties (passwords) in the various components on your canvas. If they don't match, your new NiFi will not be able to load using the flow.xml.gz file you copied over.

- Benefit of this method is you get your entire flow including all configured passwords.

2. Create a template of your entire dataflow, download it and then import it into you new NiFi.

- Provide a name and description for you template

- Once template is created, you will nee to download it:

- Click the download icon to the right of your newly created template.

- You now have a copy of your template to import in to you e new NiFi.

- Once uploaded you can then add the template to your canvas by dragging the "Template" icon to your canvas:

- Select your newly uploaded template form the selection list.

- Templates when created are sanitized of any sensitive properties so they can be used in other Nifi instances. You will need to go to any processor that used a sensitive property and re-enter those sensitive values when using this method.

Thank you,

Matt

[#2]

Thanks, Matt. I'm closer to fully understanding, however, the steps for deploying new or modified NIFI data flows to dev/qa/production is still not clear. I'm to upload templates to the production server via the canvas? Once executing, I can close my canvas window or will I eventually have multiple browser tabs of the canvas with each instance containing a different data flow? Conceptually, I'm thinking the NIFI data flow templates would be like an Oozie workflow and I would need to execute via command line with a properties file. Thanks again.

[#3]

@Blake Colson

I am not complete clear on your response. Each NiFi instance has its own UI. In a NiFi cluster no matter which node's Ui you load, the canvas you will see is that of the cluster. Every Node in a NiFi cluster must run an identical dataflow.

When you stand up your production NiFi instance or cluster, it will have its own UI and own canvas. You cannot manage your dev, qa, and prod clusters from the same UI.

Once a dataflow exists on a canvas, you can start any portion or all the components (processors, controller services, input ports, output ports, etc..). There is no dependency that the UI/canvas remain ope in your browser in order for those dataflows to continue running.

There is also no requirement that you template your entire dataflow each time. You can template only portions of a dataflow and move that portion between dev, qa, and prod. The SDLC model is not completely there yet in NiFi.

Lets say you have template dataflow version 1 and now you want to deploy version 2. Going the template route would require you to bleed out all the data traversing the current version 1 that is running.

I would upload version 2 of the template and add it to my canvas.

Then stop any ingest processors in your version 1 flow already on the canvas. Allow the remaining processors to continue to run so that all data is eventually processed out of your version 1 flow.

Start your version 2 dataflow so it starts ingesting all data at that point.

Once the version 1 flow no longer has any data queued in it any longer, you can select its components and delete them from the canvas.

Thanks,

Matt

[#4]

@Blake Colson

If you found my initial response helpful in answering your question, please accept the answer.