https://community.cloudera.com/t5/Support-Questions/Best-approach-for-getting-files-from-100-ftp-server-using/m-p/218592#M180493 <P>Hi,</P><P> I have a project that need to fetch files from over 100 ftp servers. If I use listsftp + fetchsftp approach, I will need over 100+ listsftp component for each of the ftp servers. The template will become so complicated and difficult to manage. If I use getsftp approach, I have to manage how to fetch only new files. Did anyone come across a problem like this? I need some suggestion how to handle this type of problem.</P><P> Thanks</P><P>Fan.</P> Sat, 11 Aug 2018 09:04:35 GMT dick_fan 2018-08-11T09:04:35Z https://community.cloudera.com/t5/Support-Questions/Best-approach-for-getting-files-from-100-ftp-server-using/m-p/218592#M180493 <P>Hi,</P><P> I have a project that need to fetch files from over 100 ftp servers. If I use listsftp + fetchsftp approach, I will need over 100+ listsftp component for each of the ftp servers. The template will become so complicated and difficult to manage. If I use getsftp approach, I have to manage how to fetch only new files. Did anyone come across a problem like this? I need some suggestion how to handle this type of problem.</P><P> Thanks</P><P>Fan.</P> Sat, 11 Aug 2018 09:04:35 GMT https://community.cloudera.com/t5/Support-Questions/Best-approach-for-getting-files-from-100-ftp-server-using/m-p/218592#M180493 dick_fan 2018-08-11T09:04:35Z https://community.cloudera.com/t5/Support-Questions/Best-approach-for-getting-files-from-100-ftp-server-using/m-p/218593#M180494 <P>I would start by creating a database or data file of your 100 ftp servers and credentials. Use NiFi to query this data and then send the results downstream into your data flow where downstream NiFi processors are configured dynamically using ${attributes} from the flow files generated by querying the ftp data. This will make your data flow very simple and dynamic.</P><P>If this answer helps, please choose ACCEPT.</P> Mon, 13 Aug 2018 19:55:49 GMT https://community.cloudera.com/t5/Support-Questions/Best-approach-for-getting-files-from-100-ftp-server-using/m-p/218593#M180494 stevenmatison 2018-08-13T19:55:49Z https://community.cloudera.com/t5/Support-Questions/Best-approach-for-getting-files-from-100-ftp-server-using/m-p/218594#M180495 <P>I have modified the GetSFTP Processor to accept parameters send from the attributes of the flowfile</P> Tue, 09 Oct 2018 21:37:32 GMT https://community.cloudera.com/t5/Support-Questions/Best-approach-for-getting-files-from-100-ftp-server-using/m-p/218594#M180495 dick_fan 2018-10-09T21:37:32Z