question Re: Wait for a Flowfile to be picked only after the previous flowfile has been processed completely in Support Questions

Wait for a Flowfile to be picked only after the previous flowfile has been processed completely

scoutjohn — Thu, 31 Aug 2023 09:35:12 GMT

I have a flowfile that is a list of files,

that gets split into individual flowfiles (10 in this case)

I am using FetchSFTP to fetch the file from a sftp location, the content of this fetched file is then split and is processed and stored.
In this example: I want to start processing ep.txt only after i have processed bulk.txt after that names.txt and so on.

I was referring to these links:

NiFi workflow monitoring – Wait/Notify pattern with split and merge – Pierre Villard
and
How to wait for all fragments to be processed, then do something? (ijokarumawak.github.io)

I was able to implement something like this.

but this is not what I was looking for, I want to implement a wait before fetchSFTP

I tried release signal identifier with filename , but that was stopping any flowfile from moving forward.

Please advise, thank you

Re: Wait for a Flowfile to be picked only after the previous flowfile has been processed completely

ckumar — Thu, 31 Aug 2023 12:01:28 GMT

You can evaluate the following for better approach

1. Batch use case with ListSFTP/FetchSFTP

2 . ExecuteStateless processor

Thank you

Re: Wait for a Flowfile to be picked only after the previous flowfile has been processed completely

scoutjohn — Thu, 31 Aug 2023 15:57:06 GMT

Thanks for the reply, I cannot use list sftp as it does not allow incoming request hence I have created an execute stream command processor that fetches the list.

I will then feed these list into fetch sftp and then process these files.

I want to fetch these file one by one only after the previous file has processed completely. So far with my implementation the files are picked almost immediately and processed, I cannot use the control rate processor in between because each file has different set of data and have varying processing time

Re: Wait for a Flowfile to be picked only after the previous flowfile has been processed completely

SAMSAL — Thu, 31 Aug 2023 16:15:13 GMT

Another way to consider for processing one file at a time:

https://www.youtube.com/watch?v=kvJx8vQnCNE&lc=UgzHBAI-4X3vn_dWSHV4AaABAg.9ssoo7vzq1d9ssqH-c7ttL

Re: Wait for a Flowfile to be picked only after the previous flowfile has been processed completely

scoutjohn — Mon, 04 Sep 2023 04:36:29 GMT

@SAMSAL : Thank you for this. This could be what I am looking for, Will try with this and update here.

Re: Wait for a Flowfile to be picked only after the previous flowfile has been processed completely

scoutjohn — Mon, 04 Sep 2023 05:42:11 GMT

Update :
This is working as I hoped for

With this configuration the FileProcessor group will take the next flowfile only after completely processing the flowfile that is inside the group.

Thank you @SAMSAL, @pvillard