Created 08-14-2017 09:05 PM
Hello guys! I'm new to this forum and to NiFi as well.
I'm trying to connect to an external FTP site, using ListFTP and FetchFTP processors on NiFi. The ListFTP object puts all files in a queue very fast, but the FetchFTP processor takes a lot of time to download the files, even using 5 threads on the FetchFTP it downloads 5 files and stand still for 30 seconds before starting the downloads again.
I've tried to download the same files via an external app, like Filezilla and the downloads are made in seconds.
Is there anything I can do to speed up this process?
Update:
I've tried using the GetFTP processor and it worked as fast as FileZilla. It seems that FetchFTP has some Polling Interval that isn't changeable by property.
Created 08-15-2017 12:05 PM
Are NiFi and Filezilla running on the same system or different systems?
Created 08-15-2017 01:37 PM
Yes, everything is running on the same machine.
Created 08-15-2017 06:12 PM
How is the FetchFTP processor configured?
Created on 08-16-2017 01:29 PM - edited 08-17-2019 07:46 PM
The only properties that aren't visible on this picture are hostname, port, user and password.
I've tried changing the Connection Timeout and Data Timeout, without any success.
Created 08-18-2017 06:00 PM
Try running the FetchFTP with a Completion Strategy of Delete File.
Does that make any difference?
Created 08-21-2017 05:28 PM
I've tried using Delete File as Completion Strategy but it's still slow.
Created 08-21-2017 09:58 PM
How is the Scheduling tab of the FetchFTP processor configured?
Concurrent tasks and run schedule.
Created 08-21-2017 10:22 PM
5 concurrent tasks, 1 second run schedule
Created 09-02-2017 08:15 PM
Try setting the run schedule to 0.