Limit of maximum number of processors in Nifi

[#1]

Hello Experts,

I wanted to know if there is a limit on number of processors that can be using in a Nifi Cluster. To explain further, I have a Nifi Dataflow which can only consume for a Single Kafka Topic due to transactional consistency requirements.

Now to handle consumption from multiple Kafka Topics, I have to replicate same flow (basically re-import same template) multiple times. Due to this I could potentially need 20k processors on a single Nifi cluster. Although they are grouped properly in separate process groups.

Is there a threshold on number of processors that can be used ? Or can we derive such limit based on input data volume or the kind of processors I am using.

PS: We are trying to work on using same Nifi Dataflow to cosume from multiple Topics but due to transactional sequencing and completeness requirements, it is getting over complex and might take some more time to develop. For now we have to go with replicating same flow many times.

Thanks

[#2]

Hello Haris,

we are in the same situation, in my opinion the number of processors is a hard limit in a NiFi cluster. Given the timer-driven architecture, the number of processors running at the same time is heavily influenced by the amount of threads available.

In this regard, any suggestion from NiFi developers is welcome.

For the other questions: the type of processors used and the overall flow design will also have an impact on performance, I would not consider the input volume as a problem since you can scale horizontally.

R.