Support Questions

Find answers, ask questions, and share your expertise

NiFi registry migration does not migrate the flow version dates properly

avatar
Master Collaborator

Hello,

I am trying to migrate my Nifi registry from one server to another.
Used niFi toolkit CLI for the same.

Below are the commands used -

On source registry server :
registry list-buckets
registry list-flows


On target registr server:
registry create-flow
registry sync-flow-versions


Flows with versions got migrated properly, but one issues noticed is the version date on target server.
Instead of actual date of version it shows migrated date for all versions in target server after migration.

Find the attached screnn shot for ref.

Is there any way to fix this?

 

Thanks

Mahendraimage.png

1 ACCEPTED SOLUTION

avatar
Master Mentor

@hegdemahendra 

The ./cli.sh registry sync-flow-versions toolkit description:

Syncs the versions of a flow to another flow, which could be in a different
bucket or registry. This command assumes the intent is to maintain the exact
version history across the two flows. The list of versions from the source flow
will be compared to the destination flow, and any versions not present will be
added.

 

The timestamp reflects the time which that particular flow version was added to the target flow.  In this case you are adding additional flow versions to a flow in yoru target NiFi-Registry.  All those flow versions were added at about the same time.  So the timestamp of when they were created is accurate in that target NiFi-Registry.

What this toolkit is doing is very simplistic.  You have a defined source flow (defined by a specific UUID). That flow has X number if versions.  On your target NiFi-Registry you create a new flow or have an exiting flow with its own UUID.  The flow UUIDs between source and destination will not be the same.  So these really are not the "same" flows.  Each is a unique flow, but may have the same flow versions.

Let assume if have a source flow (UUID=A) which contains 3 versions.
On my target NiFi-Registry i create a new flow (UUID=B) and I use above command to sync my 3 versions form FlowA to FlowB.  Now on some NiFi using that target NiFi-Registry imports that FlowB on to the canvas.

Now back at source, someone created versions 4, 5, and 6 for FlowA.
On that target NiFi-Registry, someone commits version 4.
Version 4 for FlowA is not the same edits as version 4 in FlowB.
Now i use same toolkit command above to again sync the Flow versions from FlowA to FlowB.  The command sees that FlowB contains versions 1-4, so flow versions 5 and 6 only are exported and imported to FlowB.    FlowB version 4 will still not match FlowA version 4.   The NiFi using FlowB will not reflect that newer versions exist and when they change versions, they will lose the changes made in FlowB version 4.

The target NiFi-Registry is properly recording the new flow version number and timestamp as each new flow version is added to FlowB (As each new flow version is added the flow version is incremented and timestamp of that import is recorded).   

As another example, you can export FlowA version 4 and import that single flow version into FlowB and it will take the next incremental flow version number of FlowB which could be version 1 if you just created FlowB.

So using the ./cli/sh registry must be done with great care.  Moving over flows initially is great, but trying to use this sync to periodically re-sync flows is likely not going to do exactly what is expected except under very controlled environments.

If your intent is to move all NiFi-Registry flows from one NiFi-Registry to another and want to preserve everything for this one time move, it is probably better to copy over the Metadata Database [1] and Flow Persistence Provider from the source NiFi-Registry to the target NiFi-Registry

[1] https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#metadata-database

[2] https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#persistence-providers

 

If you found this assisted you with your query, please take a moment to login and click "Accept" on this solution.
Thank you,

Matt

 

View solution in original post

2 REPLIES 2

avatar
Master Mentor

@hegdemahendra 

The ./cli.sh registry sync-flow-versions toolkit description:

Syncs the versions of a flow to another flow, which could be in a different
bucket or registry. This command assumes the intent is to maintain the exact
version history across the two flows. The list of versions from the source flow
will be compared to the destination flow, and any versions not present will be
added.

 

The timestamp reflects the time which that particular flow version was added to the target flow.  In this case you are adding additional flow versions to a flow in yoru target NiFi-Registry.  All those flow versions were added at about the same time.  So the timestamp of when they were created is accurate in that target NiFi-Registry.

What this toolkit is doing is very simplistic.  You have a defined source flow (defined by a specific UUID). That flow has X number if versions.  On your target NiFi-Registry you create a new flow or have an exiting flow with its own UUID.  The flow UUIDs between source and destination will not be the same.  So these really are not the "same" flows.  Each is a unique flow, but may have the same flow versions.

Let assume if have a source flow (UUID=A) which contains 3 versions.
On my target NiFi-Registry i create a new flow (UUID=B) and I use above command to sync my 3 versions form FlowA to FlowB.  Now on some NiFi using that target NiFi-Registry imports that FlowB on to the canvas.

Now back at source, someone created versions 4, 5, and 6 for FlowA.
On that target NiFi-Registry, someone commits version 4.
Version 4 for FlowA is not the same edits as version 4 in FlowB.
Now i use same toolkit command above to again sync the Flow versions from FlowA to FlowB.  The command sees that FlowB contains versions 1-4, so flow versions 5 and 6 only are exported and imported to FlowB.    FlowB version 4 will still not match FlowA version 4.   The NiFi using FlowB will not reflect that newer versions exist and when they change versions, they will lose the changes made in FlowB version 4.

The target NiFi-Registry is properly recording the new flow version number and timestamp as each new flow version is added to FlowB (As each new flow version is added the flow version is incremented and timestamp of that import is recorded).   

As another example, you can export FlowA version 4 and import that single flow version into FlowB and it will take the next incremental flow version number of FlowB which could be version 1 if you just created FlowB.

So using the ./cli/sh registry must be done with great care.  Moving over flows initially is great, but trying to use this sync to periodically re-sync flows is likely not going to do exactly what is expected except under very controlled environments.

If your intent is to move all NiFi-Registry flows from one NiFi-Registry to another and want to preserve everything for this one time move, it is probably better to copy over the Metadata Database [1] and Flow Persistence Provider from the source NiFi-Registry to the target NiFi-Registry

[1] https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#metadata-database

[2] https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#persistence-providers

 

If you found this assisted you with your query, please take a moment to login and click "Accept" on this solution.
Thank you,

Matt

 

avatar
Master Collaborator

Thanks for the great explanation Matt, much appreciated.