@Vishesh
It is not possible to say exactly what issue you may have encountered here. Do you still have the complete stack trace that followed the node disconnection exception? It is likely to have some "Caused by:.." lines in the full stack trace that may help.
Any changes being done when the disconnection occurred?
- Stopping/starting a dataflow or process group?
- Adding a template to the canvas? (if you are still using templates, they were deprecated in NiFi 1.x and Flow definitions are the new method to use. Templates were officially removed in NiFi 2.x). I have corruption issues related to templates previously.
When you restarted your service, what observations were made in the nifi-app.log on all three nodes during startup? A flow election happens first where like flows each get a vote, the flow with the most ote becomes the cluster flow and nodes without that flow will join and inherit that cluster flow. One of your three nodes would have been elected as cluster coordinator and all other nodes would have formed the cluster by sending heartbeats to that node. During that node connection phase, any node with a mismatched flow would inherit the cluster flow. Any logging related to one or more of your nodes inheriting the cluster flow on startup?
If not that then could be possible that some component was stuck in an enabling component state. So when you start a component(s) on the canvas, the component goes initially to "starting" and then "started". Likewise, stopping a component transitions to "stopping" and then "stopped". You may have been in situation where your nodes had a component stuck in the "stopping" or "starting" phase, but your cluster coordinator completed the transition. This could be caused by a bug in the component, load on the system, component has very long running process or hung process working on a FlowFile with large content, etc... Inspecting some thread dumps from those disconnected running nodes might help identifying scenario. This might be the most likely cause for you? I say this because if your flow.json.gz was corrupt, restarting your cluster would have had exceptions when trying to load the corrupted flow.json.gz. When stopping the nodes, NiFi eventually times out waiting for nodes to gracefully completed running threads and kills them. Then on restart no flow.json.gz corruption, all nodes restart fine, flow loaded successfully, and set the components to same running state.
While none of above is a definitive answer because there is not enough info to provide that, hopefully this give you an idea of what could have happened so you know what to collect or look at deeper should it happen again.
I will add that a number of fixes have gone into the newer releases (some around NiFi clustering). Apache NIFi 1.x is officially end of life. If you can not migrate to the newer Apache NiFi 2.x branch, you should at least upgrade to the latest Apache NiFi 1.28 release to take advantage of fixes done there since 1.23.
Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
