NiFi upgrade 1.9.2 to 1.23.0

[#1]

Hi All ,

I planning to upgrade NiFi 1.9.2 to 1.23.0. I read some articles about upgrading.

https://community.cloudera.com/t5/Support-Questions/Nifi-Upgrade/m-p/346309

If I follow the take special note above , can I directly migrate 1.9.2 to 1.23.0 ? Will there be any problem ?

[#2]

@Soli, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @cotopaul @SAMSAL @MattWho @TimothySpann  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

First backup your flows and settings.   i don't think you can safely upgrade that many versions.   Parameters, records, JSON format and so many other things have come.   I would leave the old one running, backup all flows, run a new clean instance with JDK 11 or 17+ with NiFi 1.23.2 and see what is not working.   https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html

See:   https://community.cloudera.com/t5/Support-Questions/Nifi-Upgrade/m-p/346309#M234832

[#4]

Hi @TimothySpann ,thanks for reply.

We use a lot of ListFile (tracking entity)+Redis cache map. We were worried about state will be losted and  cause duplicate files. Will it happend? Also, is there any recommended upgrade steps to follow for many versions migration?

[#5]

You need to work with a contractor or Cloudera support as that is four years of massive updates.   ListFile will lose tracking. Redis will remain cache (though it's in memory).   I would move away old files or grab the meta data as a checker.   It may require some updates to your coding style.

Any state that is critical to long term processing should be stored in a permanent store and use so you can move to new versions.   Say store in Postgresql or Kudu.   Kudu makes a better cache.

[#6]

 > ListFile will lose tracking.

This is beyond my knowledge. According to the introduction of ListFile, the tracking entity will be stored in redis. Why is it lost when I only upgrade nifi?

[#7 · Accepted solution]

@Soli 
There is likelihood that once you upgrade that you will have ghosted processors (happens when you are using a processor that no longer exists in the newer release) or invalid components (happens when new version of same component class introduces new required properties that need to be configured).

Something you may want to try is standing up a different NiFi on 1.23  with autoresume.state set to false in the nifi.properties. drop your flow.xml.gz in to that NiFi and start it.  Everything will load up in stopped state.  At least then you can validate al your processors are valid and none are ghosted.  If this standalone can reach yoru redis, make sure you don't start processors that use it as it can mess up state recorded in your prod NiFi.  

You also did not mention if you have a standalone NiFi or a NiFi cluster setup.  In a NiFi cluster, Zookeeper is also used by some components to store cluster wide state.  So would not utilize same ZK in your test cluster setup used to check for component issues.

At least you will be able to rule out flow design issues and be able to make note of what needs to be fixed in your production upgrade prior to doing the upgrade.
If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt