Migrating NiFi flow files between servers

[#1]

I want to move our existing data flow processes to another new server. Also, we are moving to Apache NiFi - Version 1.0.0.2.0.0.0-579 from version 1.1.1.0-12. I will appreciate advice on the best way to go about this migration.

[#2]

If you are moving a NiFi instance completely, the easiest approach could be to preserve same locations and copy over a complete directory where NiFi lives. It will pick up both flows from the conf/ directory as well as all of repositories.

[#3]

@Andrew Grande

I might not be able to keep the same locations because the new server has more specific repositories which is not what obtains in the old NiFi server.

[#4 · Accepted solution]

@Joshua Adeleke

First lets make sure we are on the same page terminology wise....

NiFi FlowFiles --> FlowFiles are made up of two parts, FlowFile Content and FlowFile Attributes. FlowFiles Content is written to NiFI's content Repository while FlowFile Attributes mostly live in JVM heap memory and the NiFi FlowFile repository. It is the FlowFile attributes that move from processor to processor in your dataflow.

Apache NiFi does not have a version 1.0.0.2.0.0.0-579. That is an HDF version of Apache NiFi 1.0.0. If you are migrating to HDF 2.0, I suggest instead migrating to HDF 2.0.1. It has many bug fixes you will want...

https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/index.html

When you say you want to move the "dataflow process: to another server, are you talking about your entire canvas and all configured processors only? Or are you also talking about moving any active FlowFiles within the old flow to the new?

My suggestion would be to stand-up your new NiFi instance/cluster. You can copy your flow.xml.gz file from your old NiFi to the new NiFi. ***NOTE: you need to make sure that the same sensitive props key is used (config setting for this is found in nifi.properties file) or your new NiFi will not start because it will not be able to decrypt any of your sensitive properties in that file. If your old NiFi was secured, you can use its authorized-users.xml file to establish the initial admin authorities in the new NiFi (configure this in authorizers.xml file). Once started you will need to access this new NiFi's UI and address any "invalid" processor/controller services, and add any controller services you may have had running on the NCM only in the old version. (No NCM in HDF 2.x versions). Some of these may be invalid because of changes to the processor/controller service properties. Once all has been addressed you are ready to move to the next step.

Shutdown all ingest processor on your old NiFi and allow it to finish processing out any data it is already working on. At the same time you can start your new NiFi NiFi so it starts ingesting any new data and begins processing it.

Thanks,

Matt

[#5]

@mclark @Timothy Spann Yes, I want to move the entire canvas with all configured processors. More like we have a new server and everything in the old now moves to the new server.

[#6]

@Joshua Adeleke

Since you are moving form two very different versions of HDF, I suggest following my procedure above. Stand-up your new HDF 2.0.1 install using a copy of your authorized-users.xml file from your old NiFi and the flow.xml.gz file from your old NiFi. Once up and running, access the UI and fix any invalid processors, controller services, and reporting tasks. There should not be many.

***NOTE: you can not copy the entire conf dir from that older version to the new as there are many changes to the files in that directory. (some do not exist in the new version and the new version has added some additional config files. What you can do is use the contents/configurations in many of the old files (named same) to configure like properties in the new NiFi's config files.

Matt

[#7]

Thank you @mclark @Timothy Spann @Andrew Grande. I eventually only needed to save and import the old templates. I forgot to mention that the HDF 2.0 had already been installed on the new server.

[#8]

https://github.com/aperepel/nifi-api-deploy

Copy all your configuration, all your conf/archive/*.gz and your conf/flow.xml.gz file.

Also from the UI, save and export all your templates separately to be sure you have them.