Configuration for sending large flowfiles via NiFi Site-to-Site transmission

[#1]

Hi

We have a situation where we have to, occasionally, send large flowfiles (>2GB) via site-to-site transmision to a NiFi canvas on another NiFi installation. Unfortunately the transmission fails for these large files. We receive messages such as:

"Awaiting transferDataLatch has been timeout"

and:

flow-files has reached to its end, but
 produced : read : wrote byte sizes ( -xxxxxxxxxx : -xxxxxxxxxx : yyyyyyyyyy) were not equal.
Something went wrong.

and on the remote machine:

"EofException: Early EOF"

It seems to me that the connection is being truncated and/or dropped before the transmission can be completed. We have tried raising the Communication Timeout on the local Remote Process Group from 30 sec to 120 sec, but it still fails. I don't know if we keep raising this it would eventually succeed, but I'm interested to know, are there some specific configurations to NiFi and/or the underlying machines that we can make that will enable these transmissions to succeed?

Many thanks

Richard

[#2 · Accepted solution]

A possible work-around might be to use a SegmentContent processor on the sending side before the RPG, and then on the receiving side use a MergeContent processor in Defragment Mode.

This would break up the file into smaller chunks on the sending send (possibly improve performance with concurrent transfers) and then reassemble it on the receiving side.

[#3]

Thanks, I think you are right - we need to focus on strategies for reducing the size of these large flowfiles rather than trying to transmit them site2site as they are.

[#4]

This should be addressed by: https://issues.apache.org/jira/browse/NIFI-5065