Hi @Benjamin Bouret Thanks for reporting this. NiFi Site-to-Site client implements different kind of timeout and expiration settings, such as cache expiration, idle connection expiration, penalization period, batch duration, and timeout. The error you shared can occur if a S2S client waited more than 'idle connection expiration'. The problem is, 'idle connection expiration' is not configurable by NiFi user at the moment. So, if data transferring takes more than the default 30 seconds, it will fail with the reported message. Even if longer 'Communication Timeout' is set at the Remote Process Group configuration. From the error message you shared, I assume you are using HTTP transport protocol for S2S. I wonder if using RAW can be a work around. But by looking at the NiFi code, it may not be the case though.. because RAW uses the 'idle connection expiration' to shutdown existing sockets, too. Split/Merge pattern will not work as you found S2S clients distribute FlowFiles among nodes in the target cluster. I think a possible work around is using other ListenXXXX processors (e.g. ListenHTTP or ListenTCP) at the target NiFi cluster. Then send data using corresponding processors such as PostHttp or PutTCP ... etc. This way, you can control how to distribute the segmented FlowFiles to target nodes. You need to do manually pick a target hostname for load balancing. It can be done with NiFi Expression Language and certain set of processors. Please refer this template: https://gist.github.com/ijokarumawak/077d7fdca57b9c8ff386f28c5198efd1 I will raise Apache NiFi JIRA so that 'idle connection expiration' can be set based on the 'Communication Timeout' value. In the meantime, I hope the above workaround works for you.
... View more
