Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Limit of maximum number of processors in Nifi


Limit of maximum number of processors in Nifi

New Contributor

Hello Experts,

I wanted to know if there is a limit on number of processors that can be using in a Nifi Cluster. To explain further, I have a Nifi Dataflow which can only consume for a Single Kafka Topic due to transactional consistency requirements.

Now to handle consumption from multiple Kafka Topics, I have to replicate same flow (basically re-import same template) multiple times. Due to this I could potentially need 20k processors on a single Nifi cluster. Although they are grouped properly in separate process groups.

Is there a threshold on number of processors that can be used ? Or can we derive such limit based on input data volume or the kind of processors I am using.

PS: We are trying to work on using same Nifi Dataflow to cosume from multiple Topics but due to transactional sequencing and completeness requirements, it is getting over complex and might take some more time to develop. For now we have to go with replicating same flow many times.