Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

Wait for a Flowfile to be picked only after the previous flowfile has been processed completely

avatar
Rising Star

I have a flowfile that is a list of files,

scoutjohn_0-1693473889872.png

that gets split into individual flowfiles (10 in this case)

I am using FetchSFTP to fetch the file from a sftp location, the content of this fetched file is then split and is processed and stored.
In this example: I want to start processing ep.txt only after i have processed bulk.txt after that names.txt and so on.


I was referring to these links:

NiFi workflow monitoring – Wait/Notify pattern with split and merge – Pierre Villard
and 
How to wait for all fragments to be processed, then do something? (ijokarumawak.github.io)

I was able to implement something like this.

scoutjohn_1-1693474216619.png

 


but this is not what I was looking for, I want to implement a wait before fetchSFTP 

scoutjohn_2-1693474309531.png


I tried release signal identifier with filename , but that was stopping any flowfile from moving forward.

 

Please advise, thank you 

1 ACCEPTED SOLUTION

avatar
Super Guru
5 REPLIES 5

avatar
Master Collaborator

You can evaluate the following for better approach 

1. Batch use case with ListSFTP/FetchSFTP

2 . ExecuteStateless processor  

Thank you 

 

avatar
Rising Star

Thanks for the reply, I cannot use list sftp as it does not allow incoming request hence I have created an execute stream command processor that fetches the list.

I will then feed  these list into fetch sftp and then process these files.

I want to fetch these file one by one only after the previous file has processed completely. So far with my implementation the files are picked almost immediately and processed, I cannot use the control rate processor in between because each file has different set of data and have varying processing time

avatar
Super Guru

avatar
Rising Star

@SAMSAL : Thank you for this. This could be what I am looking for, Will try with this and update here. 

avatar
Rising Star

Update :
This is working as I hoped for 

scoutjohn_0-1693805260874.png

scoutjohn_2-1693805976120.png

 

scoutjohn_1-1693805306230.png

With this configuration the FileProcessor group will take the next flowfile  only  after completely processing the flowfile that is inside the group.

 

Thank you @SAMSAL, @pvillard