question Re: Simple Nifi 1.6.0 Wait/Notify example in Support Questions

Simple Nifi 1.6.0 Wait/Notify example

laurie_mcintosh — Mon, 16 Apr 2018 10:55:45 GMT

To test the Nifi wait/notify processors I have the following simple example:

Getfile -> Wait -> Putfile -> Notify

So, I get some files from the getfile. I want to, one by one, put them into the putfile folder (obviously, I don't need the wait/notify processor here to do this.

So, I'd expect the getfile to get all of the files at once and hand them onto the wait. It never leaves the wait, is my problem.

i'm not getting any warnings, I have the DistributedMapCacheClientService setup as well as the DistributedMapCacheServer. I've assigned a Release Signal Identifier to the wait and notify of "release". All else is defaults.

What am I missing?

Thanks all,

---=L

Re: Simple Nifi 1.6.0 Wait/Notify example

ahadjidj — Mon, 16 Apr 2018 17:18:14 GMT

Hi @Laurie McIntosh

This is expected. Data Flows in your example are blocked at the Wait processor hence there's no flowfile going through the Putfile and then Notify to unblock the file blocked in the wait. Notify is never triggered here. You need to have your Notify in an independent flow with the triggering logic.

Thanks

Re: Simple Nifi 1.6.0 Wait/Notify example

laurie_mcintosh — Mon, 16 Apr 2018 17:29:54 GMT

Right... what I'm trying to do is setup a gate here... so the first file trips the gate (the Wait) and stops the rest flowing through, and then the first file hits the second gate (the Notify) and leaves the flow, allowing the next file to flow through. I thought the Wait/Notify pair would do that.

If not, how can I do that?

Re: Simple Nifi 1.6.0 Wait/Notify example

ahadjidj — Sun, 18 Aug 2019 02:51:29 GMT

Something you can do is to trigger a first Notify manually and let the gate do the rest. For instance, you can use a Generateflowfile with a notify for initial triggering and stop it afterward. This works but I don't know if there's a better way to do it. If the idea is to control flows, take a look at control rate processor which can be helpful.

Re: Simple Nifi 1.6.0 Wait/Notify example

laurie_mcintosh — Tue, 17 Apr 2018 02:51:32 GMT

You know when you first start coding in a language and you keep piling code on top of code to make something simple happen and it feels wrong because it shouldn't be this hard? This feels a bit like that... so I need a notify to open the gate in the first instance and then the second notify opens the gate again when the thing leaves? What if something else flows past the first notify while the process is still running? It just feels... odd. Not saying it's wrong, but it feels crufty.

What I'm looking for is something like, when count(waits) = count(notifys), let the file through.

So, at the start, count(Waits) = count(Notifys) (0 = 0), so the first file goes through. Now count(waits) = count(Notifys) + 1, so the files stop. First file leaves the flow, so Count(Waits) = count(Notifys) so next file runs.

That's a pretty specific use case, so another way to think about it is (and I'd have thought this would be a pretty standard Wait/Notify usage) is traffic lights - the "wait" is green by default and then turns red when a file passes and turns green again when the notify is tripped.

Re: Simple Nifi 1.6.0 Wait/Notify example

Mandeep_Saluja — Thu, 05 Dec 2024 17:21:51 GMT

Hi @laurie_mcintosh ,
Did you ever get around this issue?
I'm looking at the same use case in Nifi 1.18.0.

Re: Simple Nifi 1.6.0 Wait/Notify example

DianaTorres — Thu, 05 Dec 2024 21:00:06 GMT

@Mandeep_Saluja As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.

Re: Simple Nifi 1.6.0 Wait/Notify example

MattWho — Mon, 09 Dec 2024 19:53:52 GMT

@laurie_mcintosh

The Wait/Notify processors have existed a long time in Apache NiFi and are complicated to use correctly. I strongly encourage you to read the following about the Wait/Notify:

https://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/
https://pierrevillard.com/2018/2018-06-27-nifi-workflow-monitoring-wait-notify-pattern-with-split-and-merge/

With advancements in Apache NIFi and depending on your use case, there may be easier ways to handle your use case.

Process Group FlowFile Concurrency in conjunction with Process Group Outbound Policy is a newer option that has often been used in place of wait/notify setups.

Unrelated... If you are running a NiFi cluster and the target directory is a network mount directory, you should be using the ListFile/FetchFile processors in place of the GetFile.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Simple Nifi 1.6.0 Wait/Notify example

DianaTorres — Fri, 13 Dec 2024 23:59:03 GMT

@Mandeep_Saluja @laurie_mcintosh Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

Re: Simple Nifi 1.6.0 Wait/Notify example

Mandeep_Saluja — Fri, 20 Dec 2024 19:31:22 GMT

Thanks for the references @MattWho.

Currently I've implemented a workaround with counters, and querying the nifi-api with http requests to get around this.
It's definitely not a bullet proof implementation, with corner cases that need to be handled separately, but it's a start to build off of.