Processors not available in the latest Nifi Version - 1.11.4

[#1]

Hi Experts. 

We recently upgraded our Nifi Version from 1.9.2 to 1.11.4 however many of our flows arent working as some processors arent available in the new version. One of them being the "Infer Avro Schema" .

I am not sure why this should be the case as every new release should be backward compatible.  Is there a way to restore this processor onto the latest version as our code is impacted. 

We can also see that this has been an issue and raised separately as well. Please find the link here 

http://apache-nifi-users-list.2361937.n4.nabble.com/GRPC-processors-missing-after-upgrade-to-1-11-3-...

cheers

Abhinav

[#2]

@abhinav_joshi   In NiFi you can have all kinds of different processors from different versions.

The preferred procedure is to put them in a custom-libs folder location available to nifi(s), add a value to nifi config, delete the work directory in nifi(s), and restart nifi(s).  

You can see more info about how I did this on the following post:

https://community.cloudera.com/t5/Support-Questions/Can-I-put-the-NiFi-1-10-Parquet-Record-Reader-in...

[#3]

Hi , Thanks for the revert . I just did the same last night by taking up the NAR file from the previous Nifi version and placing it in the lib directory of the current Nifi version . I then went for the restart and then could see the processors. 

However , this I believe this  isn't right and every new version should be backward compatible . Yes , there can be a new flavor of a processor i.e. a few more properties here and there but not having the processor at all in the new release is actually not acceptable as it can break your existing flows . It has caused us a lot of headache as our process runs in real time . Because of this there was an outage of almost a day as it took us some time to figure out the cause and then come up with a solution . 

Is there a tech support where this can be put up so that subsequent releases should have backward compatibility . 

[#4]

You can clone the processor bundle you need from https://github.com/apache/nifi/tree/master/nifi-nar-bundles and clone it to the filearchive.

That would be solve your problem, because after every update you have the processors you need and if they came back to default, the new code will be merged in your repo.

Greets

[#5 · Accepted solution]

@abhinav_joshi 

This is something that only affects the Apache NiFi releases only.
The removal of some nars is documented in the Apache NiFi migration guidance wiki here:
https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance

"Starting in 1.10, the following nars were removed from the default convenience binary.  These include kite-nar, kafka-0-8-nar, flume-nar, media-nar, druid-controller-service-api-nar, druid-nar, other-graph-services-nar.  You can still get them from the various artifact repositories and use them in your flows but we cannot bundle them due to space limitations by default."

For Apache NiFi users, I recommend that users create a custom NiFi lib directory by adding the following property to their nifi.properties file:

nifi.nar.library.directory.<unique-string>=/<path to>/custom-lib1

for exmaple:

nifi.nar.library.directory.NiFi-1-10-nars=/nars/nifi-1-10-lib

You can add as many custom lib directories as you want.  Then place any nars that are noted as removed via the migration guidance documentation into one of these custom lib paths before upgrading.  Then as part of your upgrade process add the above property to your nifi.properties file.

Then later if additional nar bundles are deprecated, you can create another custom lib dir or just add those nars to your existing custom lib directory before upgrading.  This allows you to avoid the downtime you encountered.

The Cloudera releases of HDF and CFM include are built with all the nars by default currently as they are not affected by space limitations.

Hope this helps,

Matt

[#6]

Thanks for the revert mate. Much Appreciated.