Hi Thank you very much for this info, we will apply and let you know how it goes. Cheers   ... View more

Hi, Thank you very much for this info, we will try it and let you know how it goes. One of the other engineers was thinking of trying to start up nifi with a switch using a string that had  "FFFFFFFFFF" in it ( I only saw it briefly )  I think with the idea of forcing nifi to start with all CPUs initially. I think he was looking at also changing CPU affinity on the workstation via windows task manager but don't know where he got to with it.  Hypothetically, if you had say a setting of  nifi.flowcontroller.max.timer.driven.thread.count=36   and you had 10 processors on your canvas,  and each processor was set to run 4 concurrent tasks. In my mind this means there is either complete starvation of 1 processor downstream with zero threads and cause a bottleneck. or would Nifi reduce thread allocation to all processors to ensure it had max output like say allocating 3 threads per processor and leave 6 in reserve?  In this case would it be better to set  nifi.flowcontroller.max.timer.driven.thread.count=48 to ensure there is no thread starvation and have a safety window of 48 - (10x4) = 8 ? If you have a safety window of 8 threads, would these be allocated to say garbage collection and other activities or is that controlled by the threads in use by JVM ? Does Nifi tell the JVM it needs to run with minimum 48 threads since nifi.flowcontroller.max.timer.driven.thread.count=48 has been set  ? How does that JVM/nifi interplay work please? I'm trying to understand the right way to get tuning right to avoid resource starvation.  🙂 Cheers   ... View more

Thanks, I will digest and talk to the nifi builders about this. ... View more

Hi Matt Sorry its taken a few days to post back, we've been running tests and observing behaviour of the clusters. So we have nifi 2.5.0 on a 3 node cluster on ubuntu, and we run zookeeper as the cluster manager. We have a client Windows 11  machine with nifi 2.5.0 on it that sends data files to the cluster, the cluster processes them and sends acknowledgement files ( one each per data file ) back via Nifi to the windows machine. Its at this point where the acknowledgement files get stuck between the cluster and the client machine. We've done belt and braces baselining of zookeeper - looking at logs, making sure NTP had node times correct, memory correct, looked for errors in logs. That came up clean.  In the nifi cluster its also 3 node, and what we did last night was delete the S2S output port and that seemed to fix it. But only temporarily - the files are getting stuck again this morning.  This port connects to a down stream windows machine with nifi 2.5.0 on it. We have the port opened right up to 100,000 files and 1TB for back pressure. Penalty is 30 seconds. Traffic seems to flow reasonably evenly between all 3 nifi nodes. What we have observed and i don't know if its related, but in our non-clustered lab set up, we observed slow site to site between dockers ( 2 dockers on same ubuntu machine ), and so we had to install nifi onto linux directly which seemed to fix it. Then it slowed down again - so we deleted everything on the canvas and re-created it by importing the canvas backup JSON file we took before wiping everything .It then ran OK and seemed happy enough, with decent throughput at levels we would expect. We also observed files would "disappear" when we had 2 downstream sites connected to the same output port, we think there is a not-so-"round robin" happening, whereby we have files moving to one downstream site and others going to a second site but not in any particular sequence which makes them look like on one site they have "disappeared".  We still have files being stuck again. I'm just keen to get to the bottom of it.  🙂 We   ... View more

Thanks Matt, thats a very useful explanation. ... View more