@derisrayan Your question is impossible to answer without very detailed inspection of the following items:
- NiFi Cluster Size (# of nodes) and Spec of each Node (CPU/RAM/Disk)
- The size of the data processing per flowfile
- The number of pieces of the data arriving per execution of the flow
After the above, the configuration of the data flow for concurrency and parallelism is tuned to what NiFi Cluster performance capabilities. This comes down to Total NiFi Nodes, Total Cores, the configuration and how many active threads the NiFi Cluster can handle.
With a nicely configured NiFi cluster (3+ nodes) with as much ram and cores as possible, the transactions will be quite impressive. Scaling to 5-10-15+ nodes will increase this to an impressive production ready scale.
If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.
Thanks,
Steven @ DFHZ
