Created 08-31-2023 02:35 AM
I have a flowfile that is a list of files,
that gets split into individual flowfiles (10 in this case)
I am using FetchSFTP to fetch the file from a sftp location, the content of this fetched file is then split and is processed and stored.
In this example: I want to start processing ep.txt only after i have processed bulk.txt after that names.txt and so on.
I was referring to these links:
NiFi workflow monitoring – Wait/Notify pattern with split and merge – Pierre Villard
and
How to wait for all fragments to be processed, then do something? (ijokarumawak.github.io)
I was able to implement something like this.
but this is not what I was looking for, I want to implement a wait before fetchSFTP
I tried release signal identifier with filename , but that was stopping any flowfile from moving forward.
Please advise, thank you
Created 08-31-2023 09:15 AM
Another way to consider for processing one file at a time:
https://www.youtube.com/watch?v=kvJx8vQnCNE&lc=UgzHBAI-4X3vn_dWSHV4AaABAg.9ssoo7vzq1d9ssqH-c7ttL
Created 08-31-2023 05:01 AM
You can evaluate the following for better approach
1. Batch use case with ListSFTP/FetchSFTP
2 . ExecuteStateless processor
Thank you
Created 08-31-2023 08:57 AM
Thanks for the reply, I cannot use list sftp as it does not allow incoming request hence I have created an execute stream command processor that fetches the list.
I will then feed these list into fetch sftp and then process these files.
I want to fetch these file one by one only after the previous file has processed completely. So far with my implementation the files are picked almost immediately and processed, I cannot use the control rate processor in between because each file has different set of data and have varying processing time
Created 08-31-2023 09:15 AM
Another way to consider for processing one file at a time:
https://www.youtube.com/watch?v=kvJx8vQnCNE&lc=UgzHBAI-4X3vn_dWSHV4AaABAg.9ssoo7vzq1d9ssqH-c7ttL
Created 09-03-2023 09:36 PM
@SAMSAL : Thank you for this. This could be what I am looking for, Will try with this and update here.
Created 09-03-2023 10:42 PM