We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

[#1 · Accepted solution]

NiFi sizing is well documented here:

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_planning-your-deployment/content/ch_hard...

Here is a snippet:

In terms of configuration for best performance, take a look here:

https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.h...

In terms of deployment guild, take a look here:

https://community.hortonworks.com/articles/135337/nifi-sizing-guide-deployment-best-practices.html

[#2]

@sunile.manjee we have configuration like 256GB of RAM,12*2TB HDD,600GB SSD,dual 16 core processor for each node...my question is because of this shall we process throughput of 100000 events/sec and 100MB/s with 3 or 4 nodes ?

[#3]

@Rakesh S

-

NiFi is very capable of hitting those numbers.... But it is impossible for anyone to say base on the little information provided.
NiFi requires very little resources to run. It is the dataflows/use cases which users design and how they implement those in NiFi that impact load on the available resources of the server.

-

So while those throughput numbers may be possible in one users dataflow implementation, they may not be for another user.

The best option is to perform so load testing in your Final NiFi design to see how it performs and what resources it ends up consuming. Then tweak your dataflows design implementation to achieve better throughput.

-

Thanks,

Matt

[#4]

make sure that you install only Nifi on these nodes. Do not club Kafka and zookeeper on nifi nodes. This will imporve network i/o performance.

Dhieru