Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Configuration for sending large flowfiles via NiFi Site-to-Site transmission

avatar

Hi

We have a situation where we have to, occasionally, send large flowfiles (>2GB) via site-to-site transmision to a NiFi canvas on another NiFi installation. Unfortunately the transmission fails for these large files. We receive messages such as:

"Awaiting transferDataLatch has been timeout"

and:

flow-files has reached to its end, but
 produced : read : wrote byte sizes ( -xxxxxxxxxx : -xxxxxxxxxx : yyyyyyyyyy) were not equal.
Something went wrong.

and on the remote machine:

"EofException: Early EOF"

It seems to me that the connection is being truncated and/or dropped before the transmission can be completed. We have tried raising the Communication Timeout on the local Remote Process Group from 30 sec to 120 sec, but it still fails. I don't know if we keep raising this it would eventually succeed, but I'm interested to know, are there some specific configurations to NiFi and/or the underlying machines that we can make that will enable these transmissions to succeed?

Many thanks

Richard

1 ACCEPTED SOLUTION

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
3 REPLIES 3

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar

Thanks, I think you are right - we need to focus on strategies for reducing the size of these large flowfiles rather than trying to transmit them site2site as they are.

avatar
Super Mentor