question Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time? in Support Questions

Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

awhite4844 — Mon, 06 Mar 2017 17:42:19 GMT

Hi, i am trying to create a flow that forwards merged content only if there are 50 flowfiles in it over 1 minute. If there are less than 50, i want to drop the content. Is this possible without a custom processor?

Thank you

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

michal_rudko — Mon, 19 Aug 2019 08:22:39 GMT

Have you tried using MergeContent processor?

The whole configuration depends on how your flow file looks like, but this might be a good starting point.

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

awhite4844 — Mon, 06 Mar 2017 20:21:14 GMT

Hi @Michal R, sorry i should have been more clear, I use the MergeContent processor to merge the files. I want to know if there is a way to only forward the merged content if there are 50 files in it over a certain time. So if 50 files are merged with 1 minute then forward them, if not, drop them. I need to edit my question, sorry again.

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

michal_rudko — Mon, 06 Mar 2017 20:52:44 GMT

@adrian white Try to play around with attributes that are output by the merge content processor - merge.count and merge.bin.age. Depending on merge.bin.age value you can root the flow forward or to "/dev/null" equivalent.

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

awhite4844 — Mon, 06 Mar 2017 22:32:37 GMT

ok, thank you

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

MattWho — Mon, 19 Aug 2019 08:22:31 GMT

@adrian white

You can use the MergeContent processor followed by a RouteOnAttribute processor to accomplish what you are looking to do.

The MergeContent processor writes an attribute named "merge.count" to the FlowFile containing all your merged source flowfiles.

- Set Min entries to "50"

- Set Max Entries to "50"

- Set Max Bin age to "1 min"

The bin age timer is trigger once the very first FlowFile is added. At the end of 1 min or 50 FlowFiles (whichever occurs first) the Bin will be merged.

Connect the "merged" relationship to a RouteOnAttribute processor that checks the "merge.count" on the merged FlowFiles to verify that they contain 50 entries.

"small" will become a new relationship to the RouteOnAttribute processor.

- Auto-terminate the "small" relationship so that any merged FlowFile with fewer then 50 entries is deleted.

- Route the "unmatched" relationship on down the rest of your dataflow.

Thanks,

Matt

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

awhite4844 — Wed, 08 Mar 2017 04:49:55 GMT

Hi @Matt Clarke. Perfect, thank you