Best approach for getting files from 100+ ftp server using Listsftp/Getsftp

[#1]

Hi,

I have a project that need to fetch files from over 100 ftp servers. If I use listsftp + fetchsftp approach, I will need over 100+ listsftp component for each of the ftp servers. The template will become so complicated and difficult to manage. If I use getsftp approach, I have to manage how to fetch only new files. Did anyone come across a problem like this? I need some suggestion how to handle this type of problem.

Thanks

Fan.

[#2 · Accepted solution]

I would start by creating a database or data file of your 100 ftp servers and credentials. Use NiFi to query this data and then send the results downstream into your data flow where downstream NiFi processors are configured dynamically using ${attributes} from the flow files generated by querying the ftp data. This will make your data flow very simple and dynamic.

If this answer helps, please choose ACCEPT.

[#3]

I have modified the GetSFTP Processor to accept parameters send from the attributes of the flowfile