Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

Solved Go to solution

Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

New Contributor

Hi, i am trying to create a flow that forwards merged content only if there are 50 flowfiles in it over 1 minute. If there are less than 50, i want to drop the content. Is this possible without a custom processor?

Thank you

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

New Contributor

Have you tried using MergeContent processor?

The whole configuration depends on how your flow file looks like, but this might be a good starting point.

13268-capture.jpg

6 REPLIES 6

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

New Contributor

Have you tried using MergeContent processor?

The whole configuration depends on how your flow file looks like, but this might be a good starting point.

13268-capture.jpg

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

New Contributor

Hi @Michal R, sorry i should have been more clear, I use the MergeContent processor to merge the files. I want to know if there is a way to only forward the merged content if there are 50 files in it over a certain time. So if 50 files are merged with 1 minute then forward them, if not, drop them. I need to edit my question, sorry again.

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

New Contributor

@adrian white Try to play around with attributes that are output by the merge content processor - merge.count and merge.bin.age. Depending on merge.bin.age value you can root the flow forward or to "/dev/null" equivalent.

Highlighted

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

New Contributor

ok, thank you

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

Master Guru

@adrian white

You can use the MergeContent processor followed by a RouteOnAttribute processor to accomplish what you are looking to do.

The MergeContent processor writes an attribute named "merge.count" to the FlowFile containing all your merged source flowfiles.

13281-screen-shot-2017-03-06-at-100549-am.png

- Set Min entries to "50"

- Set Max Entries to "50"

- Set Max Bin age to "1 min"

The bin age timer is trigger once the very first FlowFile is added. At the end of 1 min or 50 FlowFiles (whichever occurs first) the Bin will be merged.

Connect the "merged" relationship to a RouteOnAttribute processor that checks the "merge.count" on the merged FlowFiles to verify that they contain 50 entries.

13282-screen-shot-2017-03-06-at-101044-am.png

"small" will become a new relationship to the RouteOnAttribute processor.

- Auto-terminate the "small" relationship so that any merged FlowFile with fewer then 50 entries is deleted.

- Route the "unmatched" relationship on down the rest of your dataflow.

Thanks,

Matt

Re: Can NiFi forward a merged content based on the number of flowfiles merged over a specified time?

New Contributor

Hi @Matt Clarke. Perfect, thank you

Don't have an account?
Coming from Hortonworks? Activate your account here