NiFi cluster processor count scalability

[#1]

Dear community,

We have a NiFi cluster installation of 3 nodes with 30GB of RAM and 15 CPUs each. The NiFi version that we are using is the 1.7.1 and we are running 43.000 NiFi processors in the cluster.

The question is if there is a known limitation of the amount of processors that we can run in the same cluster considering that we can scale horizontally and also add more CPU and memory.

Also if somebody is running more processors in the same cluster or have experience in this topic that would be a great piece of information.

Thanks a lot for any advice on this

Best Regards!

Juan Pablo

[#2 · Accepted solution]

@jpconver2   To answer your question:  No there is no actual limitation. Your experience is going to depend on number of flow files, size of payload, how many active threads, and general tuning within the flow.   For entire nifi cluster performance general tuning will depend on # nodes, min/max ram setting, number of cpu, repositiory and disk config, min/max thread count, etc.

I would say you are getting quite a lot of out 3 nodes, that is impressive.  If everything is working stable you could easily continue to add more processors until you notice stability is impacted.   I would also recommend spending some time investigating a version upgrade over 1.7.x.

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

Thanks,

Steven

[#3]

Thanks a lot for your answer!!