Simple Nifi 1.6.0 Wait/Notify example

[#1]

To test the Nifi wait/notify processors I have the following simple example:

Getfile -> Wait -> Putfile -> Notify

So, I get some files from the getfile. I want to, one by one, put them into the putfile folder (obviously, I don't need the wait/notify processor here to do this.

So, I'd expect the getfile to get all of the files at once and hand them onto the wait. It never leaves the wait, is my problem.

i'm not getting any warnings, I have the DistributedMapCacheClientService setup as well as the DistributedMapCacheServer. I've assigned a Release Signal Identifier to the wait and notify of "release". All else is defaults.

What am I missing?

Thanks all,

---=L

[#2]

Hi @Laurie McIntosh

This is expected. Data Flows in your example are blocked at the Wait processor hence there's no flowfile going through the Putfile and then Notify to unblock the file blocked in the wait. Notify is never triggered here. You need to have your Notify in an independent flow with the triggering logic.

Thanks

[#3]

Right... what I'm trying to do is setup a gate here... so the first file trips the gate (the Wait) and stops the rest flowing through, and then the first file hits the second gate (the Notify) and leaves the flow, allowing the next file to flow through. I thought the Wait/Notify pair would do that.

If not, how can I do that?

[#4]

Something you can do is to trigger a first Notify manually and let the gate do the rest. For instance, you can use a Generateflowfile with a notify for initial triggering and stop it afterward. This works but I don't know if there's a better way to do it. If the idea is to control flows, take a look at control rate processor which can be helpful.

[#5]

You know when you first start coding in a language and you keep piling code on top of code to make something simple happen and it feels wrong because it shouldn't be this hard? This feels a bit like that... so I need a notify to open the gate in the first instance and then the second notify opens the gate again when the thing leaves? What if something else flows past the first notify while the process is still running? It just feels... odd. Not saying it's wrong, but it feels crufty.

What I'm looking for is something like, when count(waits) = count(notifys), let the file through.

So, at the start, count(Waits) = count(Notifys) (0 = 0), so the first file goes through. Now count(waits) = count(Notifys) + 1, so the files stop. First file leaves the flow, so Count(Waits) = count(Notifys) so next file runs.

That's a pretty specific use case, so another way to think about it is (and I'd have thought this would be a pretty standard Wait/Notify usage) is traffic lights - the "wait" is green by default and then turns red when a file passes and turns green again when the notify is tripped.

[#6]

Hi @laurie_mcintosh , 
Did you ever get around this issue?
I'm looking at the same use case in Nifi 1.18.0.

[#7]

@Mandeep_Saluja As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.

[#8]

@laurie_mcintosh 

The Wait/Notify processors have existed a long time in Apache NiFi and are complicated to use correctly.  I strongly encourage you to read the following about the Wait/Notify:

https://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/
https://pierrevillard.com/2018/2018-06-27-nifi-workflow-monitoring-wait-notify-pattern-with-split-an...

With advancements in Apache NIFi and depending on your use case, there may be easier ways to handle your use case.  

Process Group FlowFile Concurrency in conjunction with Process Group Outbound Policy  is a newer option that has often been used in place of wait/notify setups.


Unrelated...  If you are running a NiFi cluster and the target directory is a network mount directory, you should be using the ListFile/FetchFile processors in place of the GetFile.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#9]

@Mandeep_Saluja @laurie_mcintosh Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

[#10]

Thanks for the references @MattWho.

Currently I've implemented a workaround with counters, and querying the nifi-api with http requests to get around this. 
It's definitely not a bullet proof implementation, with corner cases that need to be handled separately, but it's a start to build off of.