Does nifi offer UDP acceleration for fast large file transfer over WAN?

[#1]

Does nifi offer any UDP acceleration when transferring large files over the WAN? Something similar to Aspera's FASP or open source variants such as Tsunami UDP. I would be interested in this feature in the site to site protocol and for moving files from edge sources potentially with minifi.

[#2 · Accepted solution]

Hi @bschofield

Currently NiFi does not use any form of UDP acceleration for its site to site protocol.

One proposed solution that has been suggested previously is to add a "PutUDP" processor to match the existing "ListenUDP" processor.

The NiFi site to site protocol already batches up FlowFiles to reduce the TCP overhead during large transmissions.

Hope that helps.

[#3]

The majority of commercial products used to move files (video in the media industry) use a UDP data channel with a TCP control channel to guarantee delivery and reduce the overhead of the TCP protocol. If a PutUDP was paired with ListenUDP could/should these processors be made to use a TCP control channel to group the UDP packets? Feels like we would be going beyond the intent of what a UDP processor should do.

[#4]

@bschofield Another idea for transferring large files over a high-latency network, might be the following...

On the sending side use a SegmentContent processor to break a large FlowFile into many smaller segments, followed by a PostHTTP processor with the Concurrent Tasks increased higher than 1. This lets the sending side better utilize the network by concurrently sending segments.

On the receiving side, use a ListenHTTP processor to received the segmented FlowFiles, followed by a MergeContent processor with a Merge Strategy of Defragment. The Defragment mode will merge all the segments back together to recreate the original FlowFile.