Slow ingest on 64 core machine

[#1]

Hi

We are having problems with a single instance of nifi 2.5.0 on a physical windows 11 machine with 64 CPU cores, 64 GB RAM and NVME 12 TB ( 4 x 3 TB drives ) striped D : drive set ingesting slowly.  C: drive is for windows.

I was hoping someone could maybe give a sequential list of things to check so we at least are approaching this the right way.

The people who set up the machine have put the ingest directory ( ingest files via Getfiles ), and the nifi instance ( content repo and everything else ) on the same single D ; drive striped disk set.  Being a striped set I wouldn't have thought speed would be an issue even with both ingest and content repo on the same striped set. 

So one thing I wondered was can we force nifi to use all cores, or do we need keep some cores  ( maybe 4 ? ) for windows to use? And if we did use say 60 cores, what is the best way to do this -  and is it set at a nifi processor level,  or do you need to force Nifi to use all cores at Nifi boot up? The Nifi JVM has 8 GB of memory allocated and I haven't seen it go much past 50% utilization, which hopefully keeps garbage collection in check.

We do ingest individual 200 GB size files ( but also smaller files ) , so would it be worth re-formatting the stripe set for much larger block sizes to stop additional seeks? I'm guessing they used standard 4 KB blocks, I was thinking of maybe using 1 MB blocks?

Standard windows Antivirius is running on this machine and also our Dev machine, but in the Dev environment 8 core windows 11 machine, ingest is fast as we would expect.

Would welcome thoughts, thanks in advance....

[#2]

@zzzz77  In your bigger machine env are you adjusting the flow to tune peformance?   E.G.  Do you increase concurrency, adjust active threads pool, etc to make sure that you are getting the most possible use of the cores?    This is where you should start.    You should be able to get a lot more active threads going in the larger env before needing to worry about disk contention.  

You may want to bump up the ram min/max, but i would do this methodically.  If its 8, go 16 and see the results, then 32 and compare all 3.   32 should be as high as you need to go, but I have seen higher.  NiFi does a good job of memory management above the min/max.   

Ideally you would want nifi disks mounted separately (see docs) but since you already have a baseline in dev likely without dedicated disks,  I suspect you will see improvements using all 32+ cores vs 8 even with "slow" disks...


This is in k8s/nifi, but you will see how to crank up the CPU:

https://stevenmatison.com/blog/Max-CPU-with-NiFi-on-Minikube/

[#3]

Hi,

Thank you very much for this info, we will try it and let you know how it goes.

One of the other engineers was thinking of trying to start up nifi with a switch using a string that had  "FFFFFFFFFF" in it ( I only saw it briefly )  I think with the idea of forcing nifi to start with all CPUs initially.

I think he was looking at also changing CPU affinity on the workstation via windows task manager but don't know where he got to with it. 

Just out of interest, does Nifi start on a system with every CPU available to it on first start? 

Cheers