@Rahul Kumar Upgrading NiFi versions is not a very hard process. - 1. Install the new version in a parallel manor. for example: - lets assume 1.4 is installed at /opt/nifi/nifi-1.4.0/ - Install 1.6 in /opt/nifi/nifi1.6.0/ 2. Use the configurations from the various NiFi configuration files found in the nifi conf directory of 1.4 to configure the same named configuration files in the new 1.6 NiFi conf directory. *** It may be possible at times to simply copy the configuration files from one version to another, but be cautious here of new properties that may be added to these configuration files between releases. NiFi 1.6 should be configured to point at the same 4 repositories (database, content, flowfile, and provenance (if best practices were used these are located on separate disks outside of base NiFi install path)) being used by the old NiFi 1.4. If the This is a NiFi cluster, 1.6 should be using the same external zookeeper as the 1.4 used (If embedded ZK was used, recommend in stalling a n external ZK for new NiFi and migrating its content from embedded ZK to external ZK). New NIFi should also point to same local NiFi state directory. 3. Stop your nifi 1.4 instance/cluster. Copy the flow.xml.gz, users.xml (if secured), and authorizations.xml (if secured) from NiFi 1.4 to NiFi 1.6. 4. Start new NiFi 1.6 version. NiFi 1.6 will start, load the copied flow.xml.gz file, read in Flowfiles from the same flowfile and content repositories nifi 1.4 was using before it was shutdown. Your new NIFi will pickup processing right where the previous version left off when it was shutdown. - *** Be mindful if you have any added custom nars/jars you added tp the previous nifi version. Those will need to be moved to new Nifi as well. Recommend placing any custom nars/jars in a separate custom lib directory rather in NiFi's default lib directory. - Thank you, Matt - If you find an answer that addresses your question best, please take a moment to login to the forum and click the "Accept" link below that provided answer. ... View more

@Siddharth Sa From your image it appears that you are auto terminating the failure relationship on your putSQL processor? - Assuming the misconfiguration in your updateAttribute processor resulting in failure of every FlowFile passed to the putSQL, those FlowFiles would have all been routed to the failure relationship of the putSQL processor. It is rare that user would auto terminate a "failure" relationship as it means data is being deleted. A more typical design is to route "failure" relationships to a dumb processor that is not enabled (like an updateAttribute processor or even a funnel). This would have allowed you to redirect the connection containing that failure relationship back to your fixed updateAttribute processor resulting in all the failed data being reprocessed. - NiFi does if enable archive FlowFiles based on configured thresholds. It is possible to perform a provenance search on the FlowFiles with a "DROP" event recorded by the putSQL processor. The drop event would occur for each FlowFile routed to failure and deleted by putSQL. While not elegant, you may be able to select each failed FlowFile one by one, open the lineage, and replay the FlowFile at the "updateAttribute" point in the lineage history. You would control the sequence of processing by the older in which you replay each FlowFile. There is no bulk replay capability. - Thank you, Matt ... View more

@Raja K There are two Java process associated to a running NiFi instance. The first JVM process is tied to the NiFi bootstrap. It is what you are interacting with when you issue the "start, stop, restart, dump, etc..." During the startup phase, the bootstrap will start the other NiFi JVM process. - During startup of the main NiFi process, there is a lot that goes on. For example: - NiFi must unpack all the nars in the NiFi lib directory to the Nifi work directory and load relevant code in to memory. - NiFi must join the cluster ----- If entire cluster has just been started/restarted, NiFi goes in to an election phase (Cluster needs to elect a cluster coordinator, primary node, a cluster flow). The election will not occur until all nodes have joined (based on nifi.cluster.flow.election.max.candidates=) or until election wait time expires (nifi.cluster.flow.election.max.wait.time=5 mins). If max.candidates is not set, you will always have a min 5 min latency here. - Nodes joining the cluster must then compare their local flow with the cluster flow to make sure they match exactly. - Once the node successfully passes these steps it must start loading flowfiles back in to the flow (these would be flowfiles still queued in connection when NiFi was last stopped/restarted). - Processor components are then scheduled to run based on the cluster flow processor state (start all processors that are in a running state. Start primary node only processors on the elected primary node) - Only then is the cluster as a whole in a state where user should be given access to the UI for interactive purposes. This does not mean that the flows were not already running before theUI was actually available to the user. - There are are things that happening there, but those are the biggest pieces. Tailing the nifi-app/log watching for the "The UI is available at the following URLs:" is the easiest way to see when the UI will be available for the users. Those UIs are based on the following properties from the nifi.properties file: nifi.web.http.host= <-- unsecured nifi nifi.web.https.host= <-- secured nifi If they are left blank, NiFi will bind to all available interfaces that the JVM finds on the host machine (as your UIs shows). There is no redirection going on here. - Hope this helps explain what is occurring during the startup. The size of your dataflow(s), election process, number of queued FlowFiles being loaded back in to JVM memory (queued connections),etc. will have some impact on startup time. Your output above only has the timestamp for when the UI became available (2018-05-29 06:58:05,832), so it is not clear how much latency you are really seeing between the actual start command that timestamp. First entry in nifi-app.log should include (org.apache.nifi.NiFi Launching NiFi...). - Thank you, Matt ... View more