question Re: NiFi upgrade 1.9.2 to 1.23.0 in Support Questions

NiFi upgrade 1.9.2 to 1.23.0

Soli — Wed, 13 Sep 2023 07:33:41 GMT

Hi All ,

I planning to upgrade NiFi 1.9.2 to 1.23.0. I read some articles about upgrading.

https://community.cloudera.com/t5/Support-Questions/Nifi-Upgrade/m-p/346309

If I follow the take special note above , can I directly migrate 1.9.2 to 1.23.0 ? Will there be any problem ?

Re: NiFi upgrade 1.9.2 to 1.23.0

VidyaSargur — Wed, 13 Sep 2023 08:50:48 GMT

@Soli, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @cotopaul @SAMSAL @MattWho @TimothySpann who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: NiFi upgrade 1.9.2 to 1.23.0

TimothySpann — Wed, 13 Sep 2023 12:49:23 GMT

First backup your flows and settings. i don't think you can safely upgrade that many versions. Parameters, records, JSON format and so many other things have come. I would leave the old one running, backup all flows, run a new clean instance with JDK 11 or 17+ with NiFi 1.23.2 and see what is not working. https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html

See: https://community.cloudera.com/t5/Support-Questions/Nifi-Upgrade/m-p/346309#M234832

Re: NiFi upgrade 1.9.2 to 1.23.0

Soli — Thu, 14 Sep 2023 01:19:57 GMT

Hi @TimothySpann ,thanks for reply.

We use a lot of ListFile (tracking entity)+Redis cache map. We were worried about state will be losted and cause duplicate files. Will it happend? Also, is there any recommended upgrade steps to follow for many versions migration?

Re: NiFi upgrade 1.9.2 to 1.23.0

TimothySpann — Thu, 14 Sep 2023 12:32:33 GMT

You need to work with a contractor or Cloudera support as that is four years of massive updates. ListFile will lose tracking. Redis will remain cache (though it's in memory). I would move away old files or grab the meta data as a checker. It may require some updates to your coding style.

Any state that is critical to long term processing should be stored in a permanent store and use so you can move to new versions. Say store in Postgresql or Kudu. Kudu makes a better cache.

Re: NiFi upgrade 1.9.2 to 1.23.0

MattWho — Thu, 14 Sep 2023 17:10:50 GMT

@Soli
There is likelihood that once you upgrade that you will have ghosted processors (happens when you are using a processor that no longer exists in the newer release) or invalid components (happens when new version of same component class introduces new required properties that need to be configured).

Something you may want to try is standing up a different NiFi on 1.23 with autoresume.state set to false in the nifi.properties. drop your flow.xml.gz in to that NiFi and start it. Everything will load up in stopped state. At least then you can validate al your processors are valid and none are ghosted. If this standalone can reach yoru redis, make sure you don't start processors that use it as it can mess up state recorded in your prod NiFi.

You also did not mention if you have a standalone NiFi or a NiFi cluster setup. In a NiFi cluster, Zookeeper is also used by some components to store cluster wide state. So would not utilize same ZK in your test cluster setup used to check for component issues.

At least you will be able to rule out flow design issues and be able to make note of what needs to be fixed in your production upgrade prior to doing the upgrade.
If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: NiFi upgrade 1.9.2 to 1.23.0

MTK — Mon, 18 Sep 2023 05:27:40 GMT

> ListFile will lose tracking.

This is beyond my knowledge. According to the introduction of ListFile, the tracking entity will be stored in redis. Why is it lost when I only upgrade nifi?