question Re: Capacity planning for NiFi cluster in Archives of Support Questions (Read Only)

I agree that you should be carefully reviewing all the documented links provided by Artem Ervits, but you also need to understand the loading behavior of any given NiFI instance is directly tied to what processors are being used. While some processors exhibit little impact to CPU and/or memory, others can impact those things significantly. Capacity planning needs to take in to consideration the dataflows you want to run. What kind of data content manipulation you want to do (MergeContent, SplitContent, ReplaceContent, etc...), data sizes and volumes, how many NiFi nodes and how you plan to distributed the data load, etc...

Re: Capacity planning for NiFi cluster

shishir_saxena4 — Mon, 14 Mar 2016 20:35:32 GMT

Thank You @Artem Ervits for all the links. That was useful information.

I am still struggling with getting sizing for my requirements. I will be moving less than 100Gb of data on a daily basis so volume is not a whole lot. But my data comes in spurts. 20Mbps rate at peak times will suffice. I am wondering if I can run my Nifi client from a VM with 8 Gb of RAM and 200Gb of disk space instead of investing in a server.

Re: Capacity planning for NiFi cluster

aervits — Mon, 14 Mar 2016 20:59:37 GMT

You're welcome, see a comment above from mclark.

Re: Capacity planning for NiFi cluster

shishir_saxena4 — Mon, 14 Mar 2016 21:15:54 GMT

Thank You @mclark for surfacing this. That really makes sense.

Re: Capacity planning for NiFi cluster

MattWho — Mon, 19 Aug 2019 10:11:07 GMT

Here is a basic sizing chart for HDF:

*** But you must keep in mind that these requirements may grow depending on what processors you use in your dataflow. Memory need is often one that grows quicker then CPU need.

*** Also understand that these sizing scenarios are based upon setting up your NiFi instance(s) per the best practice documentation provided.

Re: Capacity planning for NiFi cluster

shishir_saxena4 — Tue, 15 Mar 2016 03:56:34 GMT

Thank You @mclark This is exactly what I was looking for.

Re: Capacity planning for NiFi cluster

MattWho — Thu, 17 Mar 2016 05:13:23 GMT

The NCM in a NiFi cluster typically needs more heap memory. The number of components (processors, input ports, output ports and relationships) x the number of nodes in the NiFi cluster on the graph will drive how much memory your NCM will need. For ~300 - 400 components and 3 - 4 node cluster, the NCM seems pretty good with 8GB of heap. If you encounter heap issue still, you would need to increase the heap size and/or reduce the stat buffer size and/or frequency in the nifi.properties files (NCM and Nodes).

nifi.components.status.repository.buffer.size=360 (defaults is 1440)

nifi.components.status.snapshot.frequency=5 min (default is 1)

This information is accurate as of NiFi 0.5.1 and HDF 1.1.2.