Support Questions

Find answers, ask questions, and share your expertise

Wait for a Flowfile to be picked only after the previous flowfile has been processed completely

avatar
Rising Star

I have a flowfile that is a list of files,

scoutjohn_0-1693473889872.png

that gets split into individual flowfiles (10 in this case)

I am using FetchSFTP to fetch the file from a sftp location, the content of this fetched file is then split and is processed and stored.
In this example: I want to start processing ep.txt only after i have processed bulk.txt after that names.txt and so on.


I was referring to these links:

NiFi workflow monitoring – Wait/Notify pattern with split and merge – Pierre Villard
and 
How to wait for all fragments to be processed, then do something? (ijokarumawak.github.io)

I was able to implement something like this.

scoutjohn_1-1693474216619.png

 


but this is not what I was looking for, I want to implement a wait before fetchSFTP 

scoutjohn_2-1693474309531.png


I tried release signal identifier with filename , but that was stopping any flowfile from moving forward.

 

Please advise, thank you 

1 ACCEPTED SOLUTION

avatar
Super Guru
5 REPLIES 5

avatar
Master Collaborator

You can evaluate the following for better approach 

1. Batch use case with ListSFTP/FetchSFTP

2 . ExecuteStateless processor  

Thank you 

 

avatar
Rising Star

Thanks for the reply, I cannot use list sftp as it does not allow incoming request hence I have created an execute stream command processor that fetches the list.

I will then feed  these list into fetch sftp and then process these files.

I want to fetch these file one by one only after the previous file has processed completely. So far with my implementation the files are picked almost immediately and processed, I cannot use the control rate processor in between because each file has different set of data and have varying processing time

avatar
Super Guru

avatar
Rising Star

@SAMSAL : Thank you for this. This could be what I am looking for, Will try with this and update here. 

avatar
Rising Star

Update :
This is working as I hoped for 

scoutjohn_0-1693805260874.png

scoutjohn_2-1693805976120.png

 

scoutjohn_1-1693805306230.png

With this configuration the FileProcessor group will take the next flowfile  only  after completely processing the flowfile that is inside the group.

 

Thank you @SAMSAL@pvillard