Files stuck exiting cluster

[#1]

We have Nifi 2.5.0 and a problem whereby some files seem to be "stuck" after leaving a 3 node cluster, and get "stuck" in a site-to-site connection queue after the cluster.

We ingest 60,000 small files out of a series of folders that are nested to a depth of 10 folders.

Most files seem to move through the cluster OK, but we get a few that seem to enter the cluster fine , but then have a problem leaving it. Its different files each time and they sit there for 30-60 mins and eventually move on. Its not the same files each time.

The network and compute resources are fine, the memory that Nifi is using is about 47% of the JVM maximum. Disk space doesn't seem to be an issue either. I had a look through the nifi-app.log file and nothing appears to show as an issue. Cluster heartbeat is average 10 milliseconds,  GC runs in 50 milliseconds every 30 seconds.  

Would welcome some suggestions.

Thank you in advance.

[#2]

@zzzz77 

Sorry to hear you are having challenges with your dataflow.

Can you clarify "site-to-site connection queue after the cluster" as this is not very clear.

1. Are FlowFiles always stuck in same connection queue?  Can you share a screenshot?
   
2. Is this a connection to a Remote Process Group, Remote Output Port?  Of a connection between a Remote Input Port and some other NiFi component (what is this component and how is it been configured?
3. What is the configuration settings of the connection with the "stuck/delayed" FlowFiles?

Thank you,
Matt