Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How does nifi delay scanning the files on the ftp server, thank you

avatar
Explorer

How does nifi delay scanning the files on the ftp server, which processor to choose,thank you

3 REPLIES 3

avatar
Master Collaborator

Can you please provide more details on this "delay" and why delay is needed ?

 FYI: Processors can be configured to all the time 0 sec or after a certain interval example after every 60sec or at fix time with cron driven. 

 

So if  there is chance when files may not be present on ftp to be able to Fetch by NiFI : List/Fetch/GetSFTP processors , you can schedule them specific time.

 

I can help you more once you provide more details on your use case.

 

Thank You.  

 

 

avatar
Explorer

In my case, i need use listftp processor connect ftp server, the file may be update anytime,  so i need the "Minimum File Age" property like listsftp processor get the lastast file .  i have three question,

1,do you kow how to add "Minimum File Age" property in listftp processor ?   

2, listsftp can connect ftp server ?  i try ,but faile.

3,What is the difference between listftp and listsftp?

thinks!

avatar
Master Collaborator

In response to your queries :

 

1,do you konw how to add "Minimum File Age" property in listftp processor ? 

Ans : Not supported , as ListFTP or SFTP  works on last modified timestamp to pick newly modified files since it ran before. So, if a file is added with older last modified timestamp than the one which Listftp already picked, then the file won't be picked with listftp logic. 

2.  listsftp can connect ftp server ?  i try ,but failed 

Ans : Use ListFTP is FTP server is not secure.

3. What is the difference between listftp and listsftp?  

We have two type of FTP servers, FTP and SFTP , SFTP uses a secure channel to transfer files while FTP doesn't, thus NiFi has FTP and SFTP processors. 

 

To address your use case "file may be update anytime,  so i need the "Minimum File Age" property like listsftp processor get"

 

Ans : You need to change/update the logic on how files written at FTP server , If the same files getting updated/appended multiple time assuming write is not completed yet so try to rename the file with specific name pattern after append is completed  and only list/fetch the files which matches with rename pattern using File Filter Regex settings.