We have a situation where we have to, occasionally, send large flowfiles (>2GB) via site-to-site transmision to a NiFi canvas on another NiFi installation. Unfortunately the transmission fails for these large files. We receive messages such as:
"Awaiting transferDataLatch has been timeout"
flow-files has reached to its end, but
produced : read : wrote byte sizes ( -xxxxxxxxxx : -xxxxxxxxxx : yyyyyyyyyy) were not equal.
Something went wrong.
and on the remote machine:
"EofException: Early EOF"
It seems to me that the connection is being truncated and/or dropped before the transmission can be completed. We have tried raising the Communication Timeout on the local Remote Process Group from 30 sec to 120 sec, but it still fails. I don't know if we keep raising this it would eventually succeed, but I'm interested to know, are there some specific configurations to NiFi and/or the underlying machines that we can make that will enable these transmissions to succeed?