That is because your merge content processor is running so that means merge content processor is working on those flowfiles.
if you want to list those flowfiles then stop merge content processor then only you can view them in nifi.
If you dont want to see the queue in red color then click on success relation and click on settings tab,
then configure that queue to big number and size as per your needs, by default these configurations are 10000 flowfiles or 1 GB size.
The mergeContent processor does not remove FlowFiles from an inbound connection until the actual merge process occurs. When mergeContent runs it allocates queued FlowFiles to one or more bins. While the FlowFiles themselves remain on the queue for tracking purposes, any allocated FlowFiles are now owned by that MergeContent processor. They will not be able to list or delete FlowFiles from this queue while the MergeContent processor is running.
In your case you have two NiFi nodes each with 10,000 FlowFiles for a total queue of 20,000 FlowFiles. Keep in mind that each of your nodes is merging only those FlowFiles that exist on its same node. In your current configuration you have two min set:
min number entries = 10,000 <-- that has men met on both nodes
min group size = 127 MB <-- This has not been met yet (you have about 2.2 MB on each node)
Because higher of the two must be met before a bin is eligible to merge.
Max number entries = 15,000 <-- you have not reached this on either node.
Max group size = 128 MB <-- You have not met this on either node.
NiFi will force a merge when either of these is reached. In your case, you would hit 15000 long before you reach 128 MB.
There is one more setting you have not set at all.
Max bin age =
Setting this will force the merge of a bin no matter what any of the above settings are if the age of the bin has exceed this configured value. I always recommend you set this to avoid FlowFiles lingering for exceptionally long periods of time.
You could increase the backpressure object threshold on the inbound connection, but the size you would need to in crease it to in order to hit your 127 MB min group size would amount to merging more then 565,000 FlowFiles. Since the MergeContent must hold the attributes for all FlowFiles being merged in heap memory, merging this number of FlowFiles in one operation is likely to result in out of memory errors.
In cases like this I recommend doing a two phase merge with multiple MergeContent processors.
I would configure first MergeContent to merge based on min (15000) and max (20000) number entries. You should also set a max bin age. You will need to make sure you push up your connection object threshold from 10000 to 20000.
The second MergeContent Processor would then be set to merge based on min (120 MB) and max (128 MB) group size. You should also set a max bin age.
The result of which will be far less heap usage but the same desired end result.
The flow would look something like this:
Thanks,
Matt
