@hegdemahendra  The autoload capability in NiFi can only auto-load new nars added to the directory.  It does not handle unload or reload.   The reason for this is because a reload would require the upgrade of existing components using a previously uploaded nar.  This process would require the stopping of all components added to canvas from that nar, upgrading all those components to the new nar version, and then starting the components again.  You also have the issue with the fact that the flow.json.gz has already been loaded in to memory with a different component version. Then you also have the issue of when someone adds a new nar version and does not remove the old nar version first.  You should be able to click on a component on the canvas once multiple version of same class are loaded and switch to the other version.    With the way NiFi is designed, NiFi will allow multiple versions of the same components to be loaded (always been that way).  So there has never been the capability when multiple versions of the same component are loaded to trigger an upgrade of any components from those duplicate component classes on the canvas. NiFi can only change a component's version on startup and only if only one version of the component class exists on startup.  On startup, NiFi loads the NiFi lib nars and any nars found in custom lib folders or autoload directory.  These nars get unpacked in to a work directory.  NiFi then starts loading the dataflow from the flow.json.gz file.  The flow.json.gz contains each components class, version, and configuration.  When loading a component where version is not found but ONLY one different version of that same component class is found, NiFi will switch that component that version of the class (could be older version or newer).  If any component versions changed on startup a new flow.json.gz is written out to match what is loaded in to memory. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@hegdemahendra  There currently are no NiFi processors for ingesting from Delta Lake.  There is an existing Apache Jira for creating such list/fetch processors for Delta Lake, but it is still open and does not look to be actively being worked on: https://issues.apache.org/jira/browse/NIFI-7093 I am not aware of any open source 3rd party options here as well. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@hegdemahendra  All the dataflow(s) you construct in the NiFi UI and all the controller services reporting tasks, and templates you create all reside within your NiFi's heap memory.  So the more you build the more heap that is being consumed there.  You also end up with more components being scheduled.  A scheduled processor will need a thread to check it's inbound connection or connect to an external service to check for new FlowFile or data to consume/execute upon.  All these components then need to share the available configured threads from NiFi thread pool.  A running processor is not idle.  It still has a configured run schedule to which it uses to check for work (granted that with n.  work to do those threads are extremely short lived). The default size of the Timer Driven Thread pool is only 10. So as your flow gets larger, I'd recommend looking at Garbage Collection (GC) stats for your NiFi's JVM to see how often GC is happening and how long those GC events take (All GC is stop the world, so no processing happens during GC). I'd also examine your CPU load average and is it is low, increase the size of the Max Timer Driven Thread pool (Global menu --> controller settings --> general) in small increments to see what impact that has on yoru performance.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@steven-matison - thank you.   Found out the reason for NoClassdef error it was due to the missing nar dependency in my NAR generation project pom. Need to add "nifi-mongodb-nar" dependency to make use of these mongo related classes as they are not part of standard nar. But I am already using "nifi-standard-nar" as dependency in NAR project I cant use one more, so had drop down this idea and write my own Mongo controller service.   ... View more