Migrating NiFi Registry from one server to another

[#1]

Hi all

I was looking for some help regarding moving a NiFi Registry set up from one server to another.

At the moment we set up a kind of "QA" system which was running well and so we have now created the production server if you like which we're going to use for all of our NiFi connections. The current registry is backed by a Git Repo using BitBucket which all works as expected.

We have set up the new server, installed registry and connected the same repo and all seems to be fine in terms of the NiFi servers can connect and I can see registry on the new server.

What I want to do now though, is move everything we had in the "QA" registry server, into the production server if this is possible (so that means the buckets, versions and flows etc) but JUST for the registry (we will then pull these fresh into the NiFi server(s)).

I think I read somewhere that the information in registry is kinda "linked" to a registry server via a UUID or something but I'm not sure how that works (if not, my assumption would be that everything already in the repo should just work in the registry server but it doesn't show up). Is this the case?

Is there a standard way to move something from one server to another, or a set of tools / commands that I can run to make sure that everything we have in the current registry server is in the new production server?

Many thanks in advance everyone, very much appreciated!

Craig

[#2]

@Crags 

You can not have both your NiFi-Registries linked to the same Git repository.
NiFi-Registry only pushes to the git repository.  The only time NiFi-Registry would ever read from the Git Repository is on startup.

So if you used two NiFi-Registries and  and were committing changes by both, you can cause issues with what is getting committed to your Git repo.

What is more common is to have a single NiFi-Registry which is utilized by multiple NiFi deployments.

QA NiFi builds some flow and when that flow is ready for production, it is committed to the NiFi-Registry.  That flow can the be imported from that single NiFi-Registry to the canvas of your PROD NiFi.  Now both NiFi instances are tracking to same flow in same registry.  

You then start making local changes to that same version controlled Process Group (PG) in your QA NiFi.  The PG will indicate you have local changes. you then have a couple choices on how you want to use your shared NiFi-Registry:

1. Wait until you have completed making all your changes and testing in QA before committing the next version to the shared registry.  At which time your Prod NiFi PG will indicate a newer version is now available in the shared NiFi-Registry.  You can then update your prod to that new version.
2. Incrementally commit updated versions of the PG to the shared registry.  Your prod will show new version available, so you will want to create a process for what versions are prod ready to control when a new version is actually changed in your prod.

About the UUID linkage...
Your NiFi can have one or more defined registry clients and each of those defined registry clients gets an assigned UUID on the NiFi instance (will not be same UUID on every NiFi  that sets up same registry client).  NiFi stores everything on the canvas locally in the flow.json.gz file so it can be reloaded into NiFi heap on startup.  When you start version control on a PG, the flow (gets uuid) is added to a NiFi-Registry bucket (has UUID).  Locally on the NiFi within the flow.json.gz there is now a reference to a specific NiFi-Registry client (by its UUD), a specific bucket (by its UUID) and specific flow (by its UUID).

Now considering scenario of a shared NiFi-Registry, the registry client on that NiFi will hav a different uuid even though it connects to same shared NiFi-Registry.  So using the registry client, you import a flow that flow from NiFi-Registry to the NiFi canvas.  Every component created from the import flow will get assigned UUIDs (will not match UUIDs assigned on other NiFi).  Those differences in UUIDs are not tracked as changes.

This is why if you stop version control, you can't start version control again and  connect it back to an existing flow stored in NiFi-registry.  You also can't delete the registry client and re-create it as it too would get a different UUID (NiFi blocks removing a registry client if any PG are currently using it for version control for this reason).

---------

Another option is to have a separate NiFi-Registry for each environment. When you are ready to move a flow from NiFi-Registry 1 (QA) to NiFi-Registry 2 (Prod), go into your QA NiFi-Registry, locate the flow and from the "actions" menu select export version, and select the version you want to export.

You can then go to your prod NiFi-Registry and "import new flow".  Once imported you can go to yoru Prod NiFi and load that flow onto the canvas.  

Later when you are ready in QA with a new version to push to prod, you can again export the prod ready version.  On prod NiFi-Registry, you can select the existing flow and from "actions" menu select "import new version".  This will allow you to add this flow as next version in Prod.  After doing so the version controlled PG(s) on your prod NiFi tracking against that flow will report a new version is available.

This second option allows your have better control over what changes make it to your Prod deployment.  You could also script rest-api calls to automate these steps if you wanted.

------

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3 · Accepted solution]

Hi @MattWho 

Thanks for the response, much appreciated - what I was looking at doing was simply moving the Registry from one of our servers which we had set up previously into another server which we were using for production - so not keeping 2 registries but instead using the new one and getting rid of the old one. What I didn't want to do was lose everything from the old one...

However, this turned out to be WAY easier than I thought lol

Basically we were deploying through Ansible and I had missed some configuration values and files when I was deploying which meant it was actually set to use files instead of a git repo for the storage system. Once I found this issue, updated it to the git repo and presto! all worked as expected with everything available in the new registry server from the git repository and all set up.

Thanks for the information though that's also very helpful!