Member since
02-14-2024
6
Posts
3
Kudos Received
0
Solutions
02-18-2024
06:44 PM
1 Kudo
Hi @MattWho , Thank you for the reply. I believe the FlowFile Concurrency and Outbound Policy can solve the detection problem. However, further, is there a way to run the downstream processors only once per batch of FlowFiles, as multiple FlowFiles will be released from the output port as a batch after I set Outbound Policy to Batch Output? (The downstream processor does not need to process any FlowFile but only calls some stored procedures in the database, so it should only run once per input to the overall flow.)
... View more
02-14-2024
01:27 AM
1 Kudo
Hi! I'm using NiFi 2.0.0 M1, and I'm writing a flow inside a processed group. I had a problem when I wrote the flow. The flow should be run as an individual task. Every time when it is run, it will take in some data from a table on MySQL Server, use many LookupRecord Processors and then process the matched and unmatched data respectively. Finally, it turns out to have 9 branches within the flow. After processing all these, I need to trigger an ExecuteSQLRecord processor which will call some stored procedures in MySQL. Literally, only when all the above branches finish running should this processor be triggered (and it should be triggered only once in one task). To achieve this, I wanted to use Notify and Wait processor to detect whether all the branches finish running and when I should trigger the ExecuteSQLRecord processor. However, I can only set a fixed number of signals to wait for in the Wait processor, while in the real case, sometimes not all of the 9 branches will be run. (Because, sometimes, there won't be matched or unmatched data for a lookup processor, and hence the corresponding branch won't be triggered to run.) Did anyone face the same problem as mine? Is there any good way to achieve my goal? (Below is part of the flow. You can see that I tried to use Wait but was kind of stuck there 😭)
... View more
Labels:
- Labels:
-
Apache NiFi