Slow ingest on 64 core machine

[#1]

Hi

We are having problems with a single instance of nifi 2.5.0 on a physical windows 11 machine with 64 CPU cores, 64 GB RAM and NVME 12 TB ( 4 x 3 TB drives ) striped D : drive set ingesting slowly.  C: drive is for windows.

I was hoping someone could maybe give a sequential list of things to check so we at least are approaching this the right way.

The people who set up the machine have put the ingest directory ( ingest files via Getfiles ), and the nifi instance ( content repo and everything else ) on the same single D ; drive striped disk set.  Being a striped set I wouldn't have thought speed would be an issue even with both ingest and content repo on the same striped set. 

So one thing I wondered was can we force nifi to use all cores, or do we need keep some cores  ( maybe 4 ? ) for windows to use? And if we did use say 60 cores, what is the best way to do this -  and is it set at a nifi processor level,  or do you need to force Nifi to use all cores at Nifi boot up? The Nifi JVM has 8 GB of memory allocated and I haven't seen it go much past 50% utilization, which hopefully keeps garbage collection in check.

We do ingest individual 200 GB size files ( but also smaller files ) , so would it be worth re-formatting the stripe set for much larger block sizes to stop additional seeks? I'm guessing they used standard 4 KB blocks, I was thinking of maybe using 1 MB blocks?

Standard windows Antivirius is running on this machine and also our Dev machine, but in the Dev environment 8 core windows 11 machine, ingest is fast as we would expect.

Would welcome thoughts, thanks in advance....

[#2]

@zzzz77  In your bigger machine env are you adjusting the flow to tune peformance?   E.G.  Do you increase concurrency, adjust active threads pool, etc to make sure that you are getting the most possible use of the cores?    This is where you should start.    You should be able to get a lot more active threads going in the larger env before needing to worry about disk contention.  

You may want to bump up the ram min/max, but i would do this methodically.  If its 8, go 16 and see the results, then 32 and compare all 3.   32 should be as high as you need to go, but I have seen higher.  NiFi does a good job of memory management above the min/max.   

Ideally you would want nifi disks mounted separately (see docs) but since you already have a baseline in dev likely without dedicated disks,  I suspect you will see improvements using all 32+ cores vs 8 even with "slow" disks...


This is in k8s/nifi, but you will see how to crank up the CPU:

https://stevenmatison.com/blog/Max-CPU-with-NiFi-on-Minikube/