Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Slow FTP downloads with NiFi

avatar

Hello guys! I'm new to this forum and to NiFi as well.

I'm trying to connect to an external FTP site, using ListFTP and FetchFTP processors on NiFi. The ListFTP object puts all files in a queue very fast, but the FetchFTP processor takes a lot of time to download the files, even using 5 threads on the FetchFTP it downloads 5 files and stand still for 30 seconds before starting the downloads again.

I've tried to download the same files via an external app, like Filezilla and the downloads are made in seconds.

Is there anything I can do to speed up this process?

Update:

I've tried using the GetFTP processor and it worked as fast as FileZilla. It seems that FetchFTP has some Polling Interval that isn't changeable by property.

15 REPLIES 15

avatar
@Denis Urbanavicius

Are NiFi and Filezilla running on the same system or different systems?

avatar

@Wynner

Yes, everything is running on the same machine.

avatar
@Denis Urbanavicius

How is the FetchFTP processor configured?

avatar

@Wynner

The only properties that aren't visible on this picture are hostname, port, user and password.

I've tried changing the Connection Timeout and Data Timeout, without any success.

27484-capturar.jpg

avatar
@Denis Urbanavicius

Try running the FetchFTP with a Completion Strategy of Delete File.

Does that make any difference?

avatar

I've tried using Delete File as Completion Strategy but it's still slow.

avatar
@Denis Urbanavicius

How is the Scheduling tab of the FetchFTP processor configured?

Concurrent tasks and run schedule.

avatar

5 concurrent tasks, 1 second run schedule

avatar
@Denis Urbanavicius

Try setting the run schedule to 0.