Skip to main content
zhangliang
zhangliang
Silver ⭐️⭐️
January 19, 2022
Question

How does nifi delay scanning the files on the ftp server, thank you

How does nifi delay scanning the files on the ftp server, which processor to choose,thank you

3 replies

ckumar
Platinum ⭐️
ckumar
Platinum ⭐️
January 19, 2022

Can you please provide more details on this "delay" and why delay is needed ?

 FYI: Processors can be configured to all the time 0 sec or after a certain interval example after every 60sec or at fix time with cron driven. 

 

So if  there is chance when files may not be present on ftp to be able to Fetch by NiFI : List/Fetch/GetSFTP processors , you can schedule them specific time.

 

I can help you more once you provide more details on your use case.

 

Thank You.  

 

 

Translate
    zhangliang
    zhangliangAuthor
    Silver ⭐️⭐️
    January 20, 2022

    In my case, i need use listftp processor connect ftp server, the file may be update anytime,  so i need the "Minimum File Age" property like listsftp processor get the lastast file .  i have three question,

    1,do you kow how to add "Minimum File Age" property in listftp processor ?   

    2, listsftp can connect ftp server ?  i try ,but faile.

    3,What is the difference between listftp and listsftp?

    thinks!

    Translate
    ckumar
    Platinum ⭐️
    ckumar
    Platinum ⭐️
    January 21, 2022

    In response to your queries :

     

    1,do you konw how to add "Minimum File Age" property in listftp processor ? 

    Ans : Not supported , as ListFTP or SFTP  works on last modified timestamp to pick newly modified files since it ran before. So, if a file is added with older last modified timestamp than the one which Listftp already picked, then the file won't be picked with listftp logic. 

    2.  listsftp can connect ftp server ?  i try ,but failed 

    Ans : Use ListFTP is FTP server is not secure.

    3. What is the difference between listftp and listsftp?  

    We have two type of FTP servers, FTP and SFTP , SFTP uses a secure channel to transfer files while FTP doesn't, thus NiFi has FTP and SFTP processors. 

     

    To address your use case "file may be update anytime,  so i need the "Minimum File Age" property like listsftp processor get"

     

    Ans : You need to change/update the logic on how files written at FTP server , If the same files getting updated/appended multiple time assuming write is not completed yet so try to rename the file with specific name pattern after append is completed  and only list/fetch the files which matches with rename pattern using File Filter Regex settings. 

     

     

     

     

    Translate
    Terms & ConditionsAccessibility statement