Member since
04-08-2018
6
Posts
2
Kudos Received
0
Solutions
04-16-2018
07:51 PM
1 Kudo
You know when you first start coding in a language and you keep piling code on top of code to make something simple happen and it feels wrong because it shouldn't be this hard? This feels a bit like that... so I need a notify to open the gate in the first instance and then the second notify opens the gate again when the thing leaves? What if something else flows past the first notify while the process is still running? It just feels... odd. Not saying it's wrong, but it feels crufty. What I'm looking for is something like, when count(waits) = count(notifys), let the file through. So, at the start, count(Waits) = count(Notifys) (0 = 0), so the first file goes through. Now count(waits) = count(Notifys) + 1, so the files stop. First file leaves the flow, so Count(Waits) = count(Notifys) so next file runs. That's a pretty specific use case, so another way to think about it is (and I'd have thought this would be a pretty standard Wait/Notify usage) is traffic lights - the "wait" is green by default and then turns red when a file passes and turns green again when the notify is tripped.
... View more
04-16-2018
10:29 AM
Right... what I'm trying to do is setup a gate here... so the first file trips the gate (the Wait) and stops the rest flowing through, and then the first file hits the second gate (the Notify) and leaves the flow, allowing the next file to flow through. I thought the Wait/Notify pair would do that. If not, how can I do that?
... View more
04-16-2018
03:55 AM
To test the Nifi wait/notify processors I have the following simple example: Getfile -> Wait -> Putfile -> Notify So, I get some files from the getfile. I want to, one by one, put them into the putfile folder (obviously, I don't need the wait/notify processor here to do this. So, I'd expect the getfile to get all of the files at once and hand them onto the wait. It never leaves the wait, is my problem. i'm not getting any warnings, I have the DistributedMapCacheClientService setup as well as the DistributedMapCacheServer. I've assigned a Release Signal Identifier to the wait and notify of "release". All else is defaults. What am I missing? Thanks all, ---=L
... View more
Labels:
- Labels:
-
Apache NiFi
04-14-2018
07:04 PM
Thanks Shu - difficulty with the solution as written is if I have more than one file to process. I need to: get file -> truncate table -> insert csv into table -> clean up ... all in one "transaction" as it were... so the truncation triggered by the second file doesn't happen until after the clean up for the first. Considering using some sort of semaphoring table... thoughts?
... View more
04-08-2018
05:28 AM
1 Kudo
Hi there, Fairly typical data flow requirement. CSV file, insert into mariadb staging table, do stuff to clean, import into permanent table. Got all of that working, except the bit where it needs to truncate the staging table before doing the insert. Current process is GenerateFlowData (for testing purposes) -> PutDatabaseRecord -> PutSQL (to run the post insert) I have an Execute SQL ready to go with a TRUNCATE file statement in it. I've tried inserting it between the Generate and the PutDatabaseRecord, joining on failure, but I'm not getting any data through it. Any ideas?
... View more
Labels:
- Labels:
-
Apache NiFi