Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Best approach for getting files from 100+ ftp server using Listsftp/Getsftp

Solved Go to solution
Highlighted

Best approach for getting files from 100+ ftp server using Listsftp/Getsftp

New Contributor

Hi,

I have a project that need to fetch files from over 100 ftp servers. If I use listsftp + fetchsftp approach, I will need over 100+ listsftp component for each of the ftp servers. The template will become so complicated and difficult to manage. If I use getsftp approach, I have to manage how to fetch only new files. Did anyone come across a problem like this? I need some suggestion how to handle this type of problem.

Thanks

Fan.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Best approach for getting files from 100+ ftp server using Listsftp/Getsftp

Super Collaborator

I would start by creating a database or data file of your 100 ftp servers and credentials. Use NiFi to query this data and then send the results downstream into your data flow where downstream NiFi processors are configured dynamically using ${attributes} from the flow files generated by querying the ftp data. This will make your data flow very simple and dynamic.

If this answer helps, please choose ACCEPT.

2 REPLIES 2

Re: Best approach for getting files from 100+ ftp server using Listsftp/Getsftp

Super Collaborator

I would start by creating a database or data file of your 100 ftp servers and credentials. Use NiFi to query this data and then send the results downstream into your data flow where downstream NiFi processors are configured dynamically using ${attributes} from the flow files generated by querying the ftp data. This will make your data flow very simple and dynamic.

If this answer helps, please choose ACCEPT.

Re: Best approach for getting files from 100+ ftp server using Listsftp/Getsftp

New Contributor

I have modified the GetSFTP Processor to accept parameters send from the attributes of the flowfile

Don't have an account?
Coming from Hortonworks? Activate your account here