Is there any guide available for NiFi capacity planning ? What load characteristic should be looked at ?
- Number of concurrent processors running
- Total throughput
Any help is greatly appreciated.
The NCM in a NiFi cluster typically needs more heap memory. The number of components (processors, input ports, output ports and relationships) x the number of nodes in the NiFi cluster on the graph will drive how much memory your NCM will need. For ~300 - 400 components and 3 - 4 node cluster, the NCM seems pretty good with 8GB of heap. If you encounter heap issue still, you would need to increase the heap size and/or reduce the stat buffer size and/or frequency in the nifi.properties files (NCM and Nodes).
nifi.components.status.repository.buffer.size=360 (defaults is 1440)
nifi.components.status.snapshot.frequency=5 min (default is 1)
This information is accurate as of NiFi 0.5.1 and HDF 1.1.2.