Hello, We are using a Nifi cluster with 5 nodes. Each node is a machine with 48 cores and 280-300 Gi of available memory. The issue we currently have is Nifi having trouble keeping up when a high number of tasks is required, 1,000,000+ tasks. The files are eventually transferred but errors do pop up in the mean time that seem to be irrelevant and more related to the flow trying to keep up. It's a simple flow with a GetFile > UpdateAttribute > PutSFTP > LogMesssage. I have increased concurrent tasks and batch sizes but this has little to no effect.
... View more
