@dhieru singh The mergeContent processor does not remove FlowFiles from an inbound connection until the actual merge process occurs. When mergeContent runs it allocates queued FlowFiles to one or more bins. While the FlowFiles themselves remain on the queue for tracking purposes, any allocated FlowFiles are now owned by that MergeContent processor. They will not be able to list or delete FlowFiles from this queue while the MergeContent processor is running. In your case you have two NiFi nodes each with 10,000 FlowFiles for a total queue of 20,000 FlowFiles. Keep in mind that each of your nodes is merging only those FlowFiles that exist on its same node. In your current configuration you have two min set: min number entries = 10,000 <-- that has men met on both nodes min group size = 127 MB <-- This has not been met yet (you have about 2.2 MB on each node) Because higher of the two must be met before a bin is eligible to merge. Max number entries = 15,000 <-- you have not reached this on either node. Max group size = 128 MB <-- You have not met this on either node. NiFi will force a merge when either of these is reached. In your case, you would hit 15000 long before you reach 128 MB. There is one more setting you have not set at all. Max bin age = Setting this will force the merge of a bin no matter what any of the above settings are if the age of the bin has exceed this configured value. I always recommend you set this to avoid FlowFiles lingering for exceptionally long periods of time. You could increase the backpressure object threshold on the inbound connection, but the size you would need to in crease it to in order to hit your 127 MB min group size would amount to merging more then 565,000 FlowFiles. Since the MergeContent must hold the attributes for all FlowFiles being merged in heap memory, merging this number of FlowFiles in one operation is likely to result in out of memory errors. In cases like this I recommend doing a two phase merge with multiple MergeContent processors. I would configure first MergeContent to merge based on min (15000) and max (20000) number entries. You should also set a max bin age. You will need to make sure you push up your connection object threshold from 10000 to 20000. The second MergeContent Processor would then be set to merge based on min (120 MB) and max (128 MB) group size. You should also set a max bin age. The result of which will be far less heap usage but the same desired end result. The flow would look something like this: Thanks, Matt ... View more

@Ravi Papisetti My suggestion here would be to make the following changes in your nifi.properties file on all nodes: nifi.cluster.node.protocol.threads=15 nifi.cluster.node.connection.timeout=30 secs nifi.cluster.node.read.timeout= 30 secs This will give nodes longer to respond to change requests before they get dropped by cluster coordinator. You may also want to keep an eye our for any OOM or GC issues on your nodes that may be occurring at the times of making these changes. Thanks, Matt ... View more

@Ravi Papisetti The specific mismatch is between the authorizations.xml on the disconnected node and the authorizations being used by the cluster coordinator. The Primary node plays no role in this process. You say that this happens often? Is the the same reason given every time (is it always because of authorizations mismatch)? The authorizations.xml file gets updated anytime an access policy is added, removed, or modified. For some reason the authorizations.xml file is not being updated on this node. Verify proper ownership and permission on the users.xml, authorizations.xml, and flow.xml.gz files and containing directories. The user that owns the NiFi process must be able to read, and write these files. If ownership is not an issue, you will want to check your nifi-user.log for any issues when replication requests are being made. This occurs when a change is made while logged in to any cluster node. The change must be replicated to all nodes and there may be a authentication/authorization issue preventing this node from updating. Thanks, Matt ... View more

@basant sinha * HCC Tip: Don't respond to an answer via another answer. Added comments to existing answer unless you are truly starting a new answer. Is the directory from which ListFile is listing files a mounted directory on all your nodes or only exists on just one node in your cluster? Keep in mind when using "primary node" only scheduling that the primary node can change at any time. -------- If it exists only on one node, you have a single point of failure in your cluster should that NiFi node go down. To avoid this single point of failure: 1. You could use NiFi to execute your script on "primary node" every 15 minutes assuming the directory the script is writing to is not mounted across all nodes. Then you could have ListFile running on all nodes all the time. Of course only the the listFile on any one given node will have data to ingest at any given time since your script will only be executed on the currently elected primary node. 2. Switch to using listSFTP and FetchSFTP processors, That way no matter which node is the current primary node, it can still connect over SFTP and list data. The ListSFTP processor maintains cluster wide state for this processor in Zookeeper, so when Primary node changes it does not start over from beginning. ------- The RPG is very commonly used to redistribute FlowFile within the same NiFi cluster. Check out this HCC article that covers how load-balancing occurs with a RPG: https://community.hortonworks.com/content/kbentry/109629/how-to-achieve-better-load-balancing-using-nifis-s.html Thank you, Matt ... View more

@Matt Burgess @Anders Boje NiFi-3867 is included in HDF 3.0.0 baseline: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/bk_release-notes/content/ch_hdf_relnotes.html While HDF 3.0.0 is based off Apache NiFi 1.2.0, it included many additional fixes/enhancements that later went in to Apache NiFi 1.3.0. The above release notes show all additional Apache Jiras that were included on top of everything Apache NiFi 1.2.0 that makes up the HDF 3.0.0 release. Thanks, Matt ... View more

@Shu @dhieru singh By default their is no guaranteed order in which FlowFiles are pulled from he queue feeding any given processor. This is because NiFi favor performance over order. If you want enforce some sort of order in which FlowFiles are pulled from a inbound queue, you must add a "Prioritizer" to the inbound connection. By default, no prioritizers are added. To apply a prioritizer, simply drag the desired prioritizer(s) to the "Selected Prioritizers" box. Regardless of strategy used in your DistributeLoad processor (round Robin or next available), There will not be a continuos order to the FlowFiles queued to either MergeContent processor. Thanks, Matt ... View more

@dhieru singh It may be helpful to understand your entire use case here. There is no guaranteed order in which FlowFile are merged regardless of whether one MergeContent or Multiple MergeContent processors are used. With your setup the distributeLoad processor with round robin FlowFiles from its incoming queue to its two outbound connections feeding your individual MergeContent processors. Each of those MergeContent processors will generate its own resulting merged FlowFile. One MergeContent processor with 2 concurrent tasks will perform the same as 2 MergeContent with 1 concurrent task each. If your goal here is to control heap usage by your mergeContent processors, you may want to use two MergeContent processors in series rather then in parallel. Thank you, Matt ... View more