Support Questions

Find answers, ask questions, and share your expertise

Capacity planning for NiFi cluster

avatar

Is there any guide available for NiFi capacity planning ? What load characteristic should be looked at ?

- Number of concurrent processors running

- Total throughput

Any help is greatly appreciated.

1 ACCEPTED SOLUTION

avatar
Master Mentor
11 REPLIES 11

avatar

Thank You @mclark This is exactly what I was looking for.

avatar
Master Mentor

The NCM in a NiFi cluster typically needs more heap memory. The number of components (processors, input ports, output ports and relationships) x the number of nodes in the NiFi cluster on the graph will drive how much memory your NCM will need.  For ~300 - 400 components and 3 - 4 node cluster, the NCM seems pretty good with 8GB of heap.  If you encounter heap issue still, you would need to increase the heap size and/or reduce the stat buffer size and/or frequency in the nifi.properties files (NCM and Nodes).

nifi.components.status.repository.buffer.size=360            (defaults is 1440)

nifi.components.status.snapshot.frequency=5 min          (default is 1)

This information is accurate as of NiFi 0.5.1 and HDF 1.1.2.