Nifi Rolling Update Strategy in Kubernetes (to avoid downtime)

[#1]

Hi!

Currently, we're utilizing a 3-node Nifi setup within Kubernetes, incorporating a custom NAR to integrate custom processors (approximately 30 processors developed).

When updating Nifi along with our custom processors, we typically follow these steps:

- Develop a new docker image containing the updated NAR version.
- Modify our Kubernetes deployment to incorporate the new docker image.
- Delete the Kubernetes deployment, resulting in a complete Nifi stop.
- Recreate the Kubernetes deployment, initiating Nifi with the new docker image.

Though effective, this process introduces downtime.

We're contemplating transitioning to a rolling update approach, wherein each Nifi node undergoes a stop-start cycle. However, in Nifi 1.7 at least, the node that is stopped and restarted cannot to join the cluster due to flow differences in the version of our custom processors. We're curious whether this issue persists in the latest Nifi version, say 1.25.

Another option involves deploying multiple versions of the NARs in Nifi. However, changing processor versions manually, especially considering the abundance of custom processor instances we have, presents a challenge. Ideally, in this scenario, upon restarting the pod, we'd like Nifi to automatically opt for the latest version to obviate manual intervention.
Another issue to comment is that we have a Controller Service that many of our processors use and to change the version of the listener we need to stop all processors attached to it.

So to summarize:

a) If we use only one version of the NAR and we use a rolling udpate strategy, in the last version of Nifi we will continue having the problem that a pod cannot join the cluster if it have an updated version of the processor?

b) If we use multiple NARs adn a rolling update strategy. Is there any way when we restart a nifi pod to update all our custom processors and listeners to switch to the last available version?

Thanks for any suggestions or advice!!!

Best Regards,
JP

[#2 · Accepted solution]

@jpconver2 

Challenges:

• NiFi Version: While recent versions (1.10.0+) offer improved cluster management, rolling updates can still be challenging if your custom processors introduce flow configuration changes. Nodes with the old processors won't recognize components from the updated NAR, preventing them from joining the cluster until all nodes are in sync.
• Flow Compatibility: NiFi requires consistent flow definitions (flow.xml.gz) across all nodes. Updates that alter the flow can disrupt cluster operations during rolling updates.

Solutions:

Scenario a: Single NAR Version

• Backward Compatibility: Prioritize backward compatibility in your custom processors. This ensures minimal changes to the flow definition and smoother rolling updates.
• Full Cluster Upgrade: If backward compatibility isn't feasible, consider a full cluster upgrade to the new NiFi version and custom processor NAR.

Scenario b: Multiple NAR Versions

• Manual Version Management: Update processors manually through the NiFi UI or API after deploying the new NARs. This offers control but requires intervention.
• Custom Automation Scripts: Develop scripts leveraging NiFi's REST API to automate processor version updates. These scripts can:
• Identify custom processor instances.
• Update each processor to the latest available version.
• Update controller services and restart affected processors.
• Custom NiFi Extensions: Implement custom logic to handle version upgrades. This could involve creating a Reporting Task or Controller Service that checks for new versions and updates processors automatically.

Recommendations:

• Upgrade NiFi Version: If possible, upgrade to NiFi 1.10.0 or later for improved rolling update support.
• Scripting for Automation: Explore scripting with the NiFi REST API to automate processor version updates, especially if you manage multiple NAR versions.

Remember:

• Stay updated with the latest NiFi releases to benefit from improvements and features.
• Carefully evaluate your specific needs and choose the approach that balances downtime and manageability.

Please accept it as a solution if it it helps

[#3 · Accepted solution]

@jpconver2 

Challenges:

• NiFi Version: While recent versions (1.10.0+) offer improved cluster management, rolling updates can still be challenging if your custom processors introduce flow configuration changes. Nodes with the old processors won't recognize components from the updated NAR, preventing them from joining the cluster until all nodes are in sync.
• Flow Compatibility: NiFi requires consistent flow definitions (flow.xml.gz) across all nodes. Updates that alter the flow can disrupt cluster operations during rolling updates.

Solutions:

Scenario a: Single NAR Version

• Backward Compatibility: Prioritize backward compatibility in your custom processors. This ensures minimal changes to the flow definition and smoother rolling updates.
• Full Cluster Upgrade: If backward compatibility isn't feasible, consider a full cluster upgrade to the new NiFi version and custom processor NAR.

Scenario b: Multiple NAR Versions

• Manual Version Management: Update processors manually through the NiFi UI or API after deploying the new NARs. This offers control but requires intervention.
• Custom Automation Scripts: Develop scripts leveraging NiFi's REST API to automate processor version updates. These scripts can:
• Identify custom processor instances.
• Update each processor to the latest available version.
• Update controller services and restart affected processors.
• Custom NiFi Extensions: Implement custom logic to handle version upgrades. This could involve creating a Reporting Task or Controller Service that checks for new versions and updates processors automatically.

Recommendations:

• Upgrade NiFi Version: If possible, upgrade to NiFi 1.10.0 or later for improved rolling update support.
• Scripting for Automation: Explore scripting with the NiFi REST API to automate processor version updates, especially if you manage multiple NAR versions.

Remember:

• Stay updated with the latest NiFi releases to benefit from improvements and features.
• Carefully evaluate your specific needs and choose the approach that balances downtime and manageability.

Please accept it as a solution if it it helps

[#4 · Accepted solution]

@jpconver2  @RAGHUY 

All 1.x versions of NiFi do not support rolling upgrades.  
With the major release of the NiFi 2.x versions, NiFi added rolling upgrade support as part of NIFI-12016 - Improve leniency for bundle compatibility to allow for rolling upgrades.

The above Apache NiFi jira does a great job of explaining why historically in the NiFi 1.x branch this was not implemented.  Above new feature improvement is included in NiFi 2.0.0-M1 (NiFi 2.0.0 milestone 1) or newer.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#5]

Thanks a lot for the information RAGHUY and MathWho for the comments. I will accept the solution. We have a lot of integrations built on nifi 1.X so I think we will try to find a way to overcome the problem at least for our versions and then migrate to 2.X when possible.

Thanks again!!