NiFi node disconnection from Cluster + Diff in local changes

[#1]

Hello,

Context :
- Cluster of 3
- NiFi 1.23.2 
- Registry 0.7.0 on external server

After the upgrade from 1.13.2 to 1.23.2 I have a strange reaction from the cluster of 3. I have sometime a node that disconnect with the only error that NiFi can't delete a processor "because it's destination of another component". The thing is this processor don't mean to be delete. 

My guess it's because I have a diff between local change between node in the same cluster.

For example if I voluntary create a process to create some local change, when I check on a node I have 1 local change but on another can be 501 local change. This mostly delete then add the same processor.

If I try to revert the local change to the last version of bucket, I always have a node that disconnect. So I shutdown it server side and can't start it up again because the error of desynchronisation in local flow. Even if I copy the flow.xml.gz and flow.json.gz from a healtier node, same error.

The only way I have to make it up again is to backup the flow.xml.gz/flow.json.gz and remove it from the data folder to force it generate from cluster.

Do you have any clue ? There is a file that store de local changes ?

Thanks in advance. 

[#2]

@NeheikeQ, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @SAMSAL @steven-matison @ckumar @MattWho  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

@NeheikeQ 

The first thing that stands out to me is that the version of NiFi-Registry you are using is not going to be compatible with the version of NiFi you are using.  NiFi introduced numerous new capabilities that would get tracked in NiFi-Registry, but that old NiFi-Registry version is not going to handle them (even if version control works, the stored flow definitions are going to be missing elements).

Every time a change is made on the NiFi canvas the current flow.xml.gz ad flow.json.gz files are archived and new versions created.  So rolling back can be done by swapping to the archived flow.json.gz.   There are a few bugs that have been addressed sine 1.23 that address not disconnection and failure to rejoin related bugs.  There is not enough detail here to pinpoint an exact cause for your issue. Does your issue only happen with dataflow(s) imported from your old 0.7.0 version of NiFi-Registry?  Any particular flow design that always reproduces yoru issue?  Full error and stack traces from nifi-app.log? Any other errors or warns around same time in either nifi-app.log or nifi-user.log?  What about the nifi-request.log, what was the request made at time the exception occurs?

I recommend upgrading to the latest 1.x release of Apache NiFi and to latest NIFi-Registry version to see if your issue persists. 

Note: The flow.xml.gz is deprecated. It has been replaced by the flow.json.gz.  When NiFi is started it will load the flow.json.gz.  If the flow.json.gz does not exist, NiFi will load from the flow.xml.gz file at which time it will generate the flow.json.gz file from it.  Apache NiFi 1.16+ will still write out both the flow.xml.gz and flow.json.gz files whenever a changes is made on the UI.  In Apache NiFi 2.x versions, only the flow.json.gz will exist.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#4]

Hello @MattWho,

Thanks for your quick answer.

What's make me feel it's more on NiFi side it's that we have the error on at least one maybe two environments but not the others two. Now I have succeed to revert to last version of the bucket for 2 nodes on 3 but if I up again the third one, I have temporary a question mark on the bucket itself then the asterisk without any way to display local change. When I create a process just to generate local change then I have difference between the two first node and the third one. 

We plan to upgrade the Registry after the upgrade of NiFi but as you say it's pretty mandatory. Do you think I can upgrade from Registry 0.7.0 to last one version then after upgrade NiFi to last 1.x release ?

For the log I'll check and come back to you but as I saw for the moment, nothing seem very explicit. That's why I thought about a file/folder that store local changes.

Finally, for your note, I know about the flow.xml.gz that not used anymore but as we go from 1.13.2 we still have it. Even if I copy flow.xml.gz, flow.json.gz or both of them from a working node, it won't start and error on flow desync.

Thanks.  

[#5]

@NeheikeQ 

yes, newer version of 1.x NiFi-Registry will support older versions of NiFi version controlling to it.

For NiFi after upgrade, load the flow.xml.gz on one node and start it.  Then start the other nodes so that they all inherit the flow from the one node where you had a flow.xml.gz.   At this point all nodes should join successfully and will have the same dataflow loaded. 

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt