Backup Of Nifi Cluster with 3 Nodes

[#1]

Hello All,

I'm running a Nifi cluster with 3 nodes.  Below are some questions in regards to backup. 

1. What's the best way to backup Nifi without stop \ starting?
2. Is there a way to backup all the processes running using a backup method in the event I need to restore 1 process I don't have to restore all the processes available?

Thanks!

[#2]

@jame1997 

1. Everything that is exists on the canvas resides in heap and is persisted to the flow.json.gz file written to the NiFi conf directory (default).  With every change made within the NiFi UI, the current flow.json.gz if moved to an archive directory and new flow.json.gz is generated.  All Nodes in a NiFi cluster use the same flow.json.gz.  You don;t need to stop NiFi to make a copy of the flow.json.gz file.  The NiFi flow.json.gz will contain encrypted values for all sensitive properties entered in any component on the NiFi UI canvas. In order for a NiFi instance to load a flow.json.gz, it must be cnfigured with the same "nifi.senstive.props.key" password used by the NiFi where the flow.json originated.  So always a good idea to backup your NiFi's conf directory (especially the nifi.properties file).  Keep in mind that all nodes in the NiFi Cluster must have same sensitive props key password configured, so unless you lose your entire NiFi cluster, you can get the flow.json.gz an sensitive.props.key form any nodes still accessible.

2. I am not crystal clear on what "backup all the processes" means.  Are you referring to all the dataflows built on the NiFi canvas?

Some components (processors, controller services, reporting tasks) may have dependencies on other things like local/cluster state, local files, etc.  So those items need to be taken into consideration with any backup planning.  Each node in a NiFi cluster has a local state directory which hold state for components that use local state (listFile for example).  For other components, cluster state is used (cluster state is written to Zookeeper and Zookeeper quorum maintains protection of that information).  State is always changing as your dataflow is running, so it is not something to easily backup specifically for local state that has not redundancy across NiFi nodes. 

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Hello Matt,
Thanks for the information.  With the flow.json.gz if a process group get deleted then the entire fflow.json.gz need to be restore.  I was wondering if there is another way to backup the running progress groups without taking a complete template backup that I can backup that will list the individual process groups so I can choose the one that's require a restore instead of restore all the process groups. 

Thanks!

[#4 · Accepted solution]

@jame1997 

You can version control Process Groups (PG) into NiFi-Registry.   If a process group were to get deleted, you could reload the last version stored in NiFi-Registry.  With version controlled PGs, anytime a user makes a change to the PG, the PG reports local change exists and allows a quick option to commit new version.  Not only does this version control feature allow you to restore last good stored version from NiFi-Registry, it makes it easy to back out changes to an older stored version.  This would require you to version control the individual PG and not the top level PG to create a NiFi-Registry catalog of all your PGs to facilitate easy rollback, restore, and reuse capabilities.

If you are talking about a scenario where you accidentally deleted a PG and noticed right away, you can simply swap in the newest archived flow.json.gz and restart your NiFi to restore.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt