Nifi Flow Definition does not keep the same id of processor

[#1]

I download Nifi flow definition on a Process Group. Then I search the real id of a processor and get nothing. Why the definition does not keep everything as same as those in a Nifi flow?

I need to compare two Nifi flows in two environment, DEV and UAT. But the definition does not keep all processors' identifier. It is so difficult for me to find differences.

[#2]

@wffger2 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @cotopaul @SAMSAL @MattWho  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

hi @wffger2,

I just tested on my local machine and if I download the NiFi Flow definition for a Processor Group, I can find each processor ID in that Processor Group. Are you sure you are not looking in a different export from a different environment?

If the same exact flow is imported on a different environment (meaning other servers) and you download the Flow definition, you will not have the same IDs.

For example, in your Process Group on Dev you have an Processor PutFile which has the id 1d29eb87-d66f-31ba-d653-93b704036191. If you exported everything on the UAT Environment that Processor Group and the Processor will not have the same exact IDs as on DEV because NiFi generates new UUIDs for those elements --> maybe the ID 1d29eb87-d66f-31ba-d653-93b704036191 is already in use on the UAT environment by another processor.

I might be wrong in my post and you could wait for another answer, as somebody might know otherwise.

[#4]

[#5]

@wffger2 

NiFi Flow Definitions are meant to be reusable snippets of flow.  Meaning it can be imported to the canvas of the same NiFi Cluster or to the canvas of another NiFi cluster.  Since you can not have multiple components on the canvas using the same component UUID, you will not be able to compare Flow Definitions between two different environments without seeing differences in those UUIDs.

If you want to track differences in dataflows between your UAT and DEV environments, NiFi-Registry is the method you want to use.  You can install a single NiFi-Registry and configure both your NiFi's to point at that same instance of NiFi-Registry.  Then on your DEV environment you start version control on a process group (testNiFiDefinition).  Then on your UAT environment you import that same version flow from the NiFi-registry.  Now you have both your NiFi environments tracking against the same version flow stored in that NiFi-registry.  While the component UUID will still be different, they both track to same versioned flow.  If you then make changes in DEV and commit those to NiFi-Registry as a new version of the already version controlled flow, you UAT environment will report a newer version as being available.  You can see the differences directly from the NiFi UI and easily upgrade your dataflow on the UAT environment to the newer version available if you choose.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#6 · Accepted solution]

If I may intervene here for a second, I would say that I totally agree with what @MattWho said but in the same time, I think that what you, @wffger2, are trying to achieve might be partially possible.You would still need everything what Matt said, but if you just want to double check and confirm your IDs within your definition flow with the ones from your canvas, you might achieve this goal.

So i tested with two versions right now: 1.15.3 and 1.19.1.
Now, as you can see in the first screenshot, when I downloaded the flow definition for my process group, I have at line 14 the tag "identifier", which has a different ID as the one from the NiFi Canvas. This was done on NiFi 1.15.3

If I do the same exact thing on NiFi 1.19.1, in the exported definition flow, at line 16 I have a new tag, named instanceIdentifier, which contains the exact ID of the processor, from your NiFi Canvas.

In this case, if you just want to have your ID compared to the one from your NiFi Canvas, an upgrade to a newer version might be your solution. But, pointing this again, if you want the connection between the environment, you require everything what Matt just told you.

[#7]

@wffger2 @cotopaul 
The InstanceIdentifier was introduced when NiFi switched from using the flow.xml.gz to the flow.josn.gz for flow persistence.  It is also currently being written to the flows stored in the NiFi-Registry.  I say "currently" because this jira exist and is still open:
https://issues.apache.org/jira/browse/NIFI-11321

So as @cotopaul points out, the "instanceIdentifier" will be unique for each instantiation of the flow definition, but the "identifier" will be the same.  Downside here is every time you want to do a comparison, you'll need to download latest flow definition from DEV and UAT environments and compare for difference based on the Identifiers.

Thanks,
Matt

[#8]

@wffger2 
A flow definition is a snippet of the components contained with in the Process Group (PG) on which the flow definition was exported.

The import of a flow definition is handled differently since a flow definition can be imported over and over to the same NiFi or different NiFis.  On import the components will be assigned unique new component UUIDs. So when you create/downloaded your flow definition from DEV and then imported to UAT, the UAT components will have different UUIDs.

What you should be doing is install a NiFi-Registry [1] that both your DEV and UAT environments can connect to.  This allows you to version control a Process Group (PG) on the your DEV environment and then load that version controlled PG to your UAT environment.  While the component UUIDs in UAT will still be different from DEV, both PGs will track back to same version controlled flow stored in the NiFi-Registry.  as you make changes in DEV to components in the Version controlled PG, the DEV PG will report that local changes exist.  You can commit those local changes as a new version of the PG.  At which time the same PG on your UAT env will report a newer version being available which you can change to. You will also have ability to see differences/changes between what is most recent version in NiFi-Registry and what is local to each NiFi from NiFi.

[1] https://nifi.apache.org/registry.html

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt