Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Does nifi offer UDP acceleration for fast large file transfer over WAN?

Solved Go to solution

Does nifi offer UDP acceleration for fast large file transfer over WAN?

Explorer

Does nifi offer any UDP acceleration when transferring large files over the WAN? Something similar to Aspera's FASP or open source variants such as Tsunami UDP. I would be interested in this feature in the site to site protocol and for moving files from edge sources potentially with minifi.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Does nifi offer UDP acceleration for fast large file transfer over WAN?

Hi @bschofield

Currently NiFi does not use any form of UDP acceleration for its site to site protocol.

One proposed solution that has been suggested previously is to add a "PutUDP" processor to match the existing "ListenUDP" processor.

The NiFi site to site protocol already batches up FlowFiles to reduce the TCP overhead during large transmissions.

Hope that helps.

View solution in original post

3 REPLIES 3
Highlighted

Re: Does nifi offer UDP acceleration for fast large file transfer over WAN?

Hi @bschofield

Currently NiFi does not use any form of UDP acceleration for its site to site protocol.

One proposed solution that has been suggested previously is to add a "PutUDP" processor to match the existing "ListenUDP" processor.

The NiFi site to site protocol already batches up FlowFiles to reduce the TCP overhead during large transmissions.

Hope that helps.

View solution in original post

Highlighted

Re: Does nifi offer UDP acceleration for fast large file transfer over WAN?

Explorer

The majority of commercial products used to move files (video in the media industry) use a UDP data channel with a TCP control channel to guarantee delivery and reduce the overhead of the TCP protocol. If a PutUDP was paired with ListenUDP could/should these processors be made to use a TCP control channel to group the UDP packets? Feels like we would be going beyond the intent of what a UDP processor should do.

Highlighted

Re: Does nifi offer UDP acceleration for fast large file transfer over WAN?

@bschofield Another idea for transferring large files over a high-latency network, might be the following...

On the sending side use a SegmentContent processor to break a large FlowFile into many smaller segments, followed by a PostHTTP processor with the Concurrent Tasks increased higher than 1. This lets the sending side better utilize the network by concurrently sending segments.

On the receiving side, use a ListenHTTP processor to received the segmented FlowFiles, followed by a MergeContent processor with a Merge Strategy of Defragment. The Defragment mode will merge all the segments back together to recreate the original FlowFile.

Don't have an account?
Coming from Hortonworks? Activate your account here