Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Slow FTP downloads with NiFi

Highlighted

Slow FTP downloads with NiFi

New Contributor

Hello guys! I'm new to this forum and to NiFi as well.

I'm trying to connect to an external FTP site, using ListFTP and FetchFTP processors on NiFi. The ListFTP object puts all files in a queue very fast, but the FetchFTP processor takes a lot of time to download the files, even using 5 threads on the FetchFTP it downloads 5 files and stand still for 30 seconds before starting the downloads again.

I've tried to download the same files via an external app, like Filezilla and the downloads are made in seconds.

Is there anything I can do to speed up this process?

Update:

I've tried using the GetFTP processor and it worked as fast as FileZilla. It seems that FetchFTP has some Polling Interval that isn't changeable by property.

9 REPLIES 9

Re: Slow FTP downloads with NiFi

@Denis Urbanavicius

Are NiFi and Filezilla running on the same system or different systems?

Re: Slow FTP downloads with NiFi

New Contributor

@Wynner

Yes, everything is running on the same machine.

Re: Slow FTP downloads with NiFi

@Denis Urbanavicius

How is the FetchFTP processor configured?

Re: Slow FTP downloads with NiFi

New Contributor

@Wynner

The only properties that aren't visible on this picture are hostname, port, user and password.

I've tried changing the Connection Timeout and Data Timeout, without any success.

27484-capturar.jpg

Re: Slow FTP downloads with NiFi

@Denis Urbanavicius

Try running the FetchFTP with a Completion Strategy of Delete File.

Does that make any difference?

Re: Slow FTP downloads with NiFi

New Contributor

I've tried using Delete File as Completion Strategy but it's still slow.

Re: Slow FTP downloads with NiFi

@Denis Urbanavicius

How is the Scheduling tab of the FetchFTP processor configured?

Concurrent tasks and run schedule.

Re: Slow FTP downloads with NiFi

New Contributor

5 concurrent tasks, 1 second run schedule

Re: Slow FTP downloads with NiFi

@Denis Urbanavicius

Try setting the run schedule to 0.