question Re: Does nifi offer UDP acceleration for fast large file transfer over WAN? in Archives of Support Questions (Read Only)

Does nifi offer UDP acceleration for fast large file transfer over WAN?

bschofield — Sun, 15 May 2016 19:53:25 GMT

Does nifi offer any UDP acceleration when transferring large files over the WAN? Something similar to Aspera's FASP or open source variants such as Tsunami UDP. I would be interested in this feature in the site to site protocol and for moving files from edge sources potentially with minifi.

Re: Does nifi offer UDP acceleration for fast large file transfer over WAN?

drussell — Mon, 16 May 2016 17:49:48 GMT

Hi @bschofield

Currently NiFi does not use any form of UDP acceleration for its site to site protocol.

One proposed solution that has been suggested previously is to add a "PutUDP" processor to match the existing "ListenUDP" processor.

The NiFi site to site protocol already batches up FlowFiles to reduce the TCP overhead during large transmissions.

Hope that helps.

Re: Does nifi offer UDP acceleration for fast large file transfer over WAN?

bbende — Wed, 18 May 2016 08:14:32 GMT

@bschofield Another idea for transferring large files over a high-latency network, might be the following...

On the sending side use a SegmentContent processor to break a large FlowFile into many smaller segments, followed by a PostHTTP processor with the Concurrent Tasks increased higher than 1. This lets the sending side better utilize the network by concurrently sending segments.

On the receiving side, use a ListenHTTP processor to received the segmented FlowFiles, followed by a MergeContent processor with a Merge Strategy of Defragment. The Defragment mode will merge all the segments back together to recreate the original FlowFile.

Re: Does nifi offer UDP acceleration for fast large file transfer over WAN?

bschofield — Mon, 23 May 2016 18:35:33 GMT

The majority of commercial products used to move files (video in the media industry) use a UDP data channel with a TCP control channel to guarantee delivery and reduce the overhead of the TCP protocol. If a PutUDP was paired with ListenUDP could/should these processors be made to use a TCP control channel to group the UDP packets? Feels like we would be going beyond the intent of what a UDP processor should do.