Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can I WAIT on two processors in NIFI? Final processor needs to check for both before continuing...

Solved Go to solution

How can I WAIT on two processors in NIFI? Final processor needs to check for both before continuing...

Expert Contributor

Hi, I'm working on creating a flow like this:

1st processor does work and goes into a funnel

2nd processor does work and routes to same funnel

Final processor needs to check for both before continuing. How do I accomplish this?

NOTE: I do not need the actual file flows, they are only used as trigger files.

Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How can I WAIT on two processors in NIFI? Final processor needs to check for both before continuing...

Rising Star

@jpetro416

Another possible solution until the Wait and Notify processors are available.

You can use MergeContent configured to only merge when it hits 2 files (which may work for you because you don't need the final flow files and you just want a trigger in this case):

13828-nifi-wait-trigger.png

This first flow trigger will be held in queue before the MergeContent, and will only be merged once the second flow trigger has arrived. The merged flowfile that comes out of the MergeContent will be your final trigger.

Note, I used the following configurations to get this to work:

  • Back pressure on both 'success' flows going into MergeContent (set to only allow 1 file in queue at a time before the merge)
  • MergeContent configured with:
    • Minimum Number of Entries = 2
    • Maximum Number of Entries = 2
    • Correlation Attribute Name = 'my-attribute' (I added this attribute to each UpdateAttribute processor - this would only be required if you wanted to have multiple triggers going into the MergeContent and you wanted to control the final trigger based on a correlation attribute)

I've attached the xml flow template for your reference.

mergecontent-trigger-wait.xml

4 REPLIES 4

Re: How can I WAIT on two processors in NIFI? Final processor needs to check for both before continuing...

I think you could might be able to use the new Wait and Notify processors that should be in the upcoming 1.2 release of Apache NiFi.

1st processor -> Notify -> Funnel

2nd processor -> Notify -> Funnel

Funnel -> Wait (Signal Count == 2)

Not sure if that is exactly what you described, but seems like it could work.

https://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/

Re: How can I WAIT on two processors in NIFI? Final processor needs to check for both before continuing...

Expert Contributor

Bump for any other possible solutions?

Highlighted

Re: How can I WAIT on two processors in NIFI? Final processor needs to check for both before continuing...

Rising Star

@jpetro416

Another possible solution until the Wait and Notify processors are available.

You can use MergeContent configured to only merge when it hits 2 files (which may work for you because you don't need the final flow files and you just want a trigger in this case):

13828-nifi-wait-trigger.png

This first flow trigger will be held in queue before the MergeContent, and will only be merged once the second flow trigger has arrived. The merged flowfile that comes out of the MergeContent will be your final trigger.

Note, I used the following configurations to get this to work:

  • Back pressure on both 'success' flows going into MergeContent (set to only allow 1 file in queue at a time before the merge)
  • MergeContent configured with:
    • Minimum Number of Entries = 2
    • Maximum Number of Entries = 2
    • Correlation Attribute Name = 'my-attribute' (I added this attribute to each UpdateAttribute processor - this would only be required if you wanted to have multiple triggers going into the MergeContent and you wanted to control the final trigger based on a correlation attribute)

I've attached the xml flow template for your reference.

mergecontent-trigger-wait.xml

Re: How can I WAIT on two processors in NIFI? Final processor needs to check for both before continuing...

Expert Contributor

Thank you, this is exactly the type of workaround I was looking for!

Don't have an account?
Coming from Hortonworks? Activate your account here