Created 01-18-2022 10:54 PM
How does nifi delay scanning the files on the ftp server, which processor to choose,thank you
Created 01-19-2022 04:58 AM
Can you please provide more details on this "delay" and why delay is needed ?
FYI: Processors can be configured to all the time 0 sec or after a certain interval example after every 60sec or at fix time with cron driven.
So if there is chance when files may not be present on ftp to be able to Fetch by NiFI : List/Fetch/GetSFTP processors , you can schedule them specific time.
I can help you more once you provide more details on your use case.
Thank You.
Created 01-19-2022 05:06 PM
In my case, i need use listftp processor connect ftp server, the file may be update anytime, so i need the "Minimum File Age" property like listsftp processor get the lastast file . i have three question,
1,do you kow how to add "Minimum File Age" property in listftp processor ?
2, listsftp can connect ftp server ? i try ,but faile.
3,What is the difference between listftp and listsftp?
thinks!
Created on 01-21-2022 03:08 AM - edited 01-21-2022 03:09 AM
In response to your queries :
1,do you konw how to add "Minimum File Age" property in listftp processor ?
Ans : Not supported , as ListFTP or SFTP works on last modified timestamp to pick newly modified files since it ran before. So, if a file is added with older last modified timestamp than the one which Listftp already picked, then the file won't be picked with listftp logic.
2. listsftp can connect ftp server ? i try ,but failed
Ans : Use ListFTP is FTP server is not secure.
3. What is the difference between listftp and listsftp?
We have two type of FTP servers, FTP and SFTP , SFTP uses a secure channel to transfer files while FTP doesn't, thus NiFi has FTP and SFTP processors.
To address your use case "file may be update anytime, so i need the "Minimum File Age" property like listsftp processor get"
Ans : You need to change/update the logic on how files written at FTP server , If the same files getting updated/appended multiple time assuming write is not completed yet so try to rename the file with specific name pattern after append is completed and only list/fetch the files which matches with rename pattern using File Filter Regex settings.