question Re: How does nifi delay scanning the files on the ftp server, thank you in Support Questions

How does nifi delay scanning the files on the ftp server, thank you

zhangliang — Wed, 19 Jan 2022 06:54:27 GMT

How does nifi delay scanning the files on the ftp server, which processor to choose,thank you

Re: How does nifi delay scanning the files on the ftp server, thank you

ckumar — Wed, 19 Jan 2022 12:58:17 GMT

Can you please provide more details on this "delay" and why delay is needed ?

FYI: Processors can be configured to all the time 0 sec or after a certain interval example after every 60sec or at fix time with cron driven.

So if there is chance when files may not be present on ftp to be able to Fetch by NiFI : List/Fetch/GetSFTP processors , you can schedule them specific time.

I can help you more once you provide more details on your use case.

Thank You.

Re: How does nifi delay scanning the files on the ftp server, thank you

zhangliang — Thu, 20 Jan 2022 01:06:01 GMT

In my case, i need use listftp processor connect ftp server, the file may be update anytime, so i need the "Minimum File Age" property like listsftp processor get the lastast file . i have three question,

1,do you kow how to add "Minimum File Age" property in listftp processor ?

2, listsftp can connect ftp server ? i try ,but faile.

3,What is the difference between listftp and listsftp?

thinks!

Re: How does nifi delay scanning the files on the ftp server, thank you

ckumar — Fri, 21 Jan 2022 11:09:17 GMT

In response to your queries :

1,do you konw how to add "Minimum File Age" property in listftp processor ?

Ans : Not supported , as ListFTP or SFTP works on last modified timestamp to pick newly modified files since it ran before. So, if a file is added with older last modified timestamp than the one which Listftp already picked, then the file won't be picked with listftp logic.

2. listsftp can connect ftp server ? i try ,but failed

Ans : Use ListFTP is FTP server is not secure.

3. What is the difference between listftp and listsftp?

We have two type of FTP servers, FTP and SFTP , SFTP uses a secure channel to transfer files while FTP doesn't, thus NiFi has FTP and SFTP processors.

To address your use case "file may be update anytime, so i need the "Minimum File Age" property like listsftp processor get"

Ans : You need to change/update the logic on how files written at FTP server , If the same files getting updated/appended multiple time assuming write is not completed yet so try to rename the file with specific name pattern after append is completed and only list/fetch the files which matches with rename pattern using File Filter Regex settings.