Issue in nifi upgrade from 1.23.2 to 2.3.0

[#1]

Hi Team,
Below are the basic steps that I am following to upgrade Nifi from version 1.23.2 to version 2.3.0 on my RHEL 9 server:

unzip nifi newer version: unzip nifi-2.3.0-bin.zip

copying flow files from older to newer version: cp nifi-1.23.2/conf/flow.* nifi-2.3.0/conf/

copying state directory from older to newer version: cp -r nifi-1.23.2/state nifi-2.3.0/state/

update nifi properties for sensitive prop key: nifi.sensitive.props.key=<<same as older version>>

start nifi: sh nifi-2.3.0/bin/nifi.sh start

Problem Statement:
I am getting the Nifi UI console for newer verion (Nifi-2.3.0). All the processGroups Or Processors those were in older verion copied to new version as well. But few processors (like joltTransformation, publishKafka and etc), it still showing that those are using 1.23.2 nar file.

Troubleshooting steps, but did not work:

1. Though I have deleted the work directory and revalidated that lib is not using any nar file of older version.
   • Delete work directory: rm -rf nifi-2.3.0/work/*
   • Delete lib directory with 1.23.2 in case if it has: rm -rf nifi-2.3.0/lib/*.1.23.2*.nar
   • restart nifi service: sh nifi-2.3.0/bin/nifi.sh restart
2. I even manually changed the version in flow.json and flow.xml file from 1.23.2 to 2.3.0. Doing this the libbrary is 2.3.0 now, but in invalid state.

Can you please mention the steps that I am missing here while upgrade from 1.23.2 to 2.3.0? Why not all the processoris getting upgraded as expected?

[#2]

@PriyankaMondal 

There significant differences between the Apache NiFi 1.x and Apache NiFi 2.x major release. 

• Deprecated and removed processors, controller services, and reporting task components
• Some components moved to new nars
• Deprecated and removed NiFi Templates
• Deprecated and removed NiFi Variable registry.

This means that you can not simply move your flow.json.gz from Apache NiFi 1.23.2 to Apache NiFi 2.x.  First you should update your dataflows so they are no longer using any of the deprecated components in Apache NiFi 1.x.  I recommend first upgrading to Apache NiFi 1.28 so you have the latest deprecation logging.  Apache NiFi 1.28 should produce a deprecation log that will tell you all the deprecated components you are currently using on your dataflow canvas.  Take steps to remove these components or replace them with alternative components that are still available in Apache NiFi 2.x.

Examples: 

• JoltTransformJson processor was included in the  nifi-standard-nar in Apache NiFi 1.x , but has moved to a nifi-jolt-nar on Apache NiFi  2.x.  So the class name has changed.  You'll need to add the Apache NiFi 2.x class version of JoltTransformJson processor to canvas and reconfigure  it for your dataflow and delete the ghosted (dashed line around it because NiFi does not know that class)  JoltTransformJson processor.  While you manually changed the version in yoru flow.json.gz, you did not change the class path as needed resulting in the processor still being ghosted.
• ConvertAvroToJson processor was deprecated in NiFi 1.x and removed in Apache NiFi 2.x.  Would need to replace it with a convertRecord (available in Apache NIFi 1.x and 2.x) configured to use an Avro Reader and Json Writer.
• NiFi Variable Registry removed.  So if you are using any NiFi variables in your processor configuration in Apache NiFi 1.x, you'll need to modify your dataflows to use NiFi parameters instead (Parameters exist in Apache NiFi1.x and 2.x)
• Templates were deprecate in NiFi 1.x and were replaced with Flow Definitions. Templates removed in NiFi 2.x. Would need to remove all templates saved in NiFi before moving to Apache NiFi 2.x.

Above is just a short list. refer to the deprecation log produced by Apache NiFi 1.28 to see all deprecated processor you may have been using in yoru dataflows.

I do wish there was an easier way to move from Apache NiFi 1.x to 2.x, but depending on your use of deprecated features and changed component classes, there may be little too a lot of effort needed to prepare yoru NiFi 1.x datfalows for migration to NiFi 2.x

For Cloudera Flow Management license holders:

For Cloudera Flow Management NiFi users, Cloudera has built a Cloudera Flow Management Migration Tool that automated many of the migration steps (swapping processors when alternatives exist, changing processor classes to new classes, converting templates to flow definitions, converting NiFi variables to NiFi parameters, etc.  While there is still no direct upgrade possible from Cloudera Flow Management 2.1.7 (Apache NiFi 1.x based) to Cloudera Flow Management 4.10 (Apache NIFi 2.x based), this migration tool takes a lot of the manual work out of preparing your flow.json.gz for the new major release.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt