Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

[#1]

Hi, i am trying to create a flow that forwards merged content only if there are 50 flowfiles in it over 1 minute. If there are less than 50, i want to drop the content. Is this possible without a custom processor?

Thank you

[#2 · Accepted solution]

Have you tried using MergeContent processor?

The whole configuration depends on how your flow file looks like, but this might be a good starting point.

[#3]

Hi @Michal R, sorry i should have been more clear, I use the MergeContent processor to merge the files. I want to know if there is a way to only forward the merged content if there are 50 files in it over a certain time. So if 50 files are merged with 1 minute then forward them, if not, drop them. I need to edit my question, sorry again.

[#4]

@adrian white Try to play around with attributes that are output by the merge content processor - merge.count and merge.bin.age. Depending on merge.bin.age value you can root the flow forward or to "/dev/null" equivalent.

[#5]

ok, thank you

[#6]

@adrian white

You can use the MergeContent processor followed by a RouteOnAttribute processor to accomplish what you are looking to do.

The MergeContent processor writes an attribute named "merge.count" to the FlowFile containing all your merged source flowfiles.

- Set Min entries to "50"

- Set Max Entries to "50"

- Set Max Bin age to "1 min"

The bin age timer is trigger once the very first FlowFile is added. At the end of 1 min or 50 FlowFiles (whichever occurs first) the Bin will be merged.

Connect the "merged" relationship to a RouteOnAttribute processor that checks the "merge.count" on the merged FlowFiles to verify that they contain 50 entries.

"small" will become a new relationship to the RouteOnAttribute processor.

- Auto-terminate the "small" relationship so that any merged FlowFile with fewer then 50 entries is deleted.

- Route the "unmatched" relationship on down the rest of your dataflow.

Thanks,

Matt

[#7]

Hi @Matt Clarke. Perfect, thank you