How can I save my NiFi flows into XML?

[#1]

I need to save my NiFi flow into an XML file.

We can import the xml file into template, but I also need to save NiFi flows as an XML file

[#2 · Accepted solution]

If you select a section of your flow, you can turn that into a template with the New Template button

You can then use the template manager:

This lists all your templates and allows you to download them as an XML file.

If you want to save the entire flow you have in the system, you can also find that in nifi/conf/flow.xml.gz on your nifi box. This is not a template, but would be able to drop into a clean NiFi instance.

[#3]

@Manikandan Durairaj

Simon is completely correct above; however, I want to add a little to his statement about saving the entire flow.xml.gz (Standalone or NiFI Cluster Node) file or flow.tar (NiFi Cluster NCM) file. When you generate templates in NiFi, those dataflows are scrubbed of all encrypted values (passwords). When importing those templates in to another NiFi, the user will need to repopulate all the processor and controller tasks passwords manually. Saving off the flow.xml.gz or flow.tar file will capture the entire flow exactly as it is, encrypted sensitive passwords and all. NiFi will not start if it cannot decrypt these encrypted sensitive properties contained in the flow.xml. When sensitive properties (passwords) are added they are encrypted using these settings from your nifi.properties file:

# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC

In order to drop your entire flow.xml.gz or flow.tar onto another clean NiFi, these values must all match exactly. Thanks,

Matt

[#4]

Hi @Matt Clarke, I tried coping flow.xml.gz and nifi.properties security setting but after that NiFi is crashing with below error, any suggestions?

ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.IllegalArgumentException: Unable to access properties loader in the expected manner - apparent classpath or build issue
java.lang.IllegalArgumentException: Unable to access properties loader in the expected manner - apparent classpath or build issue.

[#5]

Hi @Matt Clarke, I tried coping flow.xml.gz and nifi.properties security setting but after that NiFi is crashing with below error, any suggestions?

ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.IllegalArgumentException: Unable to access properties loader in the expected manner - apparent classpath or build issue
java.lang.IllegalArgumentException: Unable to access properties loader in the expected manner - apparent classpath or build issue

[#6]

Looking at our current environment, seems like there are 2 instances of flow.xml.gz file. One under nifi-node and another under nifi-cluster-manager. Which file do we need to copy over ? Both?

[#7]

In a healthy cluster they should both be the same.

[#8]

Ok. The question is if we rehydrate to a new cluster should we copy over the flow files to both locations?

[#9]

Looking at our current environment, seems like there are 2 instances of flow.xml.gz file. One under nifi-node and another under nifi-cluster-manager. Which file do we need to copy over ? Both?

[#10]

Looking at our current environment, seems like there are 2 instances of flow.xml.gz file. One under nifi-node and another under nifi-cluster-manager. Which file do we need to copy over ? Both?

[#11]

@sagar gaikwad

Difference between templating the entire NiFi canvas and copying the actual flow.xml.* files. When you generate a template it is saved as and xml and contains all the configuration from the canvas with the EXCEPTION of sensitive property values (passwords). Sensitive processor properties are encrypted using the sensitive props key configured in the nifi.properties file. So by removing these values when a template is created the templates to be loaded into any other NiFi successfully. If you capture the flow.xml.* files directly from NiFi, you will also need to save/reserve the value set for the sensitive props key in order to use them on another system. The above is true no matter which version of NiFi you are using.

NiFi 0.x /HDF 1.x versions:

With these versions of NiFi you have a NiFi Cluster Manager (NCM) and NiFi nodes.

The Nodes all have identical flow.xml.gz files and an identical list of templates inside the /templates directory.

The NCM has a flow.tar file. This file is not the same as what is on the nodes. The flow.tar consists of numerous files important to the NCM only. (NCM specific controller services, primary node designator, etc..) The flow.tar also includes a flow.xml file. It is from this flow.xml file on the NCM that the flow.xml.gz and templates inside the /templates dir on the nodes are derived from.

You can build a new cluster if you have the flow.tar file as it contains everything new nodes need plus what ta new NCM needs. The flow.tar cannot be used on a standalone install of NiFi nor can it be used by the nodes directly themselves. It is intended for an NCM only.

The flow.xml.gz file from any of the nodes can be used to stand-up a standalone NiFi instance.

You may choose to keep a backup of both.

NiFi 1.x /HDF 2.x versions:

These versions of NiFi have eliminated the need for a NCM (zero-master clustering). As a result every instance of Nifi is a node and they all have identical flow.xml.gz files. These flow.xml.gz files do not contain the exact same content as the nodes from 0.x versions of NiFi. The templates have all been moved into the flow.xml.gz file and there is no longer a need for a separate /templates dir.

With this version you can keep a backup or flow.xml.gz from any node. The flow.xml.gz file can be used to buidl another cluster or a standalone NiFi instance.

Thanks,

Matt

[#12]

Side note: You can install a NiFi 1.x or HDF 2.x and use the flow.xml.gz file from an older 0.5.x+ or HDF 1.1+ version of NiFi. The newer version of NiFi will convert it to the new format.

[#13]

If I create a new template, it is creating a flow.xml.gz and working fine. However If I replace the flow.xml.gz with old flow.xml.gz (one backup taken earlier in same cluster node HDF2.X) nifi UI is not coming after login and giving error as "com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out".

Have tried with all parameter tuning as described by "Matt Clarke" in some other post but no results.

Again If I move old file and replace with new flow.xml.gz then nifi is working fine.

Please let me know if anyone faces such issues and probable reason and working around.

Thanks,

Suman