question Re: Configuration for sending large flowfiles via NiFi Site-to-Site transmission in Archives of Support Questions (Read Only)

Configuration for sending large flowfiles via NiFi Site-to-Site transmission

richard_d_corfi — Tue, 13 Mar 2018 21:08:33 GMT

We have a situation where we have to, occasionally, send large flowfiles (>2GB) via site-to-site transmision to a NiFi canvas on another NiFi installation. Unfortunately the transmission fails for these large files. We receive messages such as:

"Awaiting transferDataLatch has been timeout"

and:

flow-files has reached to its end, but
 produced : read : wrote byte sizes ( -xxxxxxxxxx : -xxxxxxxxxx : yyyyyyyyyy) were not equal.
Something went wrong.

and on the remote machine:

"EofException: Early EOF"

It seems to me that the connection is being truncated and/or dropped before the transmission can be completed. We have tried raising the Communication Timeout on the local Remote Process Group from 30 sec to 120 sec, but it still fails. I don't know if we keep raising this it would eventually succeed, but I'm interested to know, are there some specific configurations to NiFi and/or the underlying machines that we can make that will enable these transmissions to succeed?

Many thanks

Richard

Re: Configuration for sending large flowfiles via NiFi Site-to-Site transmission

bbende — Wed, 14 Mar 2018 00:20:02 GMT

A possible work-around might be to use a SegmentContent processor on the sending side before the RPG, and then on the receiving side use a MergeContent processor in Defragment Mode.

This would break up the file into smaller chunks on the sending send (possibly improve performance with concurrent transfers) and then reassemble it on the receiving side.

Re: Configuration for sending large flowfiles via NiFi Site-to-Site transmission

richard_d_corfi — Wed, 14 Mar 2018 16:44:22 GMT

Thanks, I think you are right - we need to focus on strategies for reducing the size of these large flowfiles rather than trying to transmit them site2site as they are.

Re: Configuration for sending large flowfiles via NiFi Site-to-Site transmission

MattWho — Tue, 03 Jul 2018 23:50:04 GMT

This should be addressed by: https://issues.apache.org/jira/browse/NIFI-5065