Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How can I WAIT on two processors in NIFI? Final processor needs to check for both before continuing...

avatar
Expert Contributor

Hi, I'm working on creating a flow like this:

1st processor does work and goes into a funnel

2nd processor does work and routes to same funnel

Final processor needs to check for both before continuing. How do I accomplish this?

NOTE: I do not need the actual file flows, they are only used as trigger files.

Thanks!

1 ACCEPTED SOLUTION

avatar
Expert Contributor

@jpetro416

Another possible solution until the Wait and Notify processors are available.

You can use MergeContent configured to only merge when it hits 2 files (which may work for you because you don't need the final flow files and you just want a trigger in this case):

13828-nifi-wait-trigger.png

This first flow trigger will be held in queue before the MergeContent, and will only be merged once the second flow trigger has arrived. The merged flowfile that comes out of the MergeContent will be your final trigger.

Note, I used the following configurations to get this to work:

  • Back pressure on both 'success' flows going into MergeContent (set to only allow 1 file in queue at a time before the merge)
  • MergeContent configured with:
    • Minimum Number of Entries = 2
    • Maximum Number of Entries = 2
    • Correlation Attribute Name = 'my-attribute' (I added this attribute to each UpdateAttribute processor - this would only be required if you wanted to have multiple triggers going into the MergeContent and you wanted to control the final trigger based on a correlation attribute)

I've attached the xml flow template for your reference.

mergecontent-trigger-wait.xml

View solution in original post

4 REPLIES 4

avatar
Master Guru

I think you could might be able to use the new Wait and Notify processors that should be in the upcoming 1.2 release of Apache NiFi.

1st processor -> Notify -> Funnel

2nd processor -> Notify -> Funnel

Funnel -> Wait (Signal Count == 2)

Not sure if that is exactly what you described, but seems like it could work.

https://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/

avatar
Expert Contributor

Bump for any other possible solutions?

avatar
Expert Contributor

@jpetro416

Another possible solution until the Wait and Notify processors are available.

You can use MergeContent configured to only merge when it hits 2 files (which may work for you because you don't need the final flow files and you just want a trigger in this case):

13828-nifi-wait-trigger.png

This first flow trigger will be held in queue before the MergeContent, and will only be merged once the second flow trigger has arrived. The merged flowfile that comes out of the MergeContent will be your final trigger.

Note, I used the following configurations to get this to work:

  • Back pressure on both 'success' flows going into MergeContent (set to only allow 1 file in queue at a time before the merge)
  • MergeContent configured with:
    • Minimum Number of Entries = 2
    • Maximum Number of Entries = 2
    • Correlation Attribute Name = 'my-attribute' (I added this attribute to each UpdateAttribute processor - this would only be required if you wanted to have multiple triggers going into the MergeContent and you wanted to control the final trigger based on a correlation attribute)

I've attached the xml flow template for your reference.

mergecontent-trigger-wait.xml

avatar
Expert Contributor

Thank you, this is exactly the type of workaround I was looking for!