question Re: We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering in Archives of Support Questions (Read Only)

We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

rakesh_satheesa — Wed, 11 Jul 2018 11:46:43 GMT

Re: We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

sunile_manjee — Sun, 18 Aug 2019 09:00:49 GMT

NiFi sizing is well documented here:

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_planning-your-deployment/content/ch_hardware-sizing.html

Here is a snippet:

In terms of configuration for best performance, take a look here:

https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html

In terms of deployment guild, take a look here:

https://community.hortonworks.com/articles/135337/nifi-sizing-guide-deployment-best-practices.html

Re: We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

rakesh_satheesa — Wed, 11 Jul 2018 12:19:09 GMT

@sunile.manjee we have configuration like 256GB of RAM,12*2TB HDD,600GB SSD,dual 16 core processor for each node...my question is because of this shall we process throughput of 100000 events/sec and 100MB/s with 3 or 4 nodes ?

Re: We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

MattWho — Thu, 12 Jul 2018 00:34:55 GMT

@Rakesh S

NiFi is very capable of hitting those numbers.... But it is impossible for anyone to say base on the little information provided.
NiFi requires very little resources to run. It is the dataflows/use cases which users design and how they implement those in NiFi that impact load on the available resources of the server.

So while those throughput numbers may be possible in one users dataflow implementation, they may not be for another user.

The best option is to perform so load testing in your Final NiFi design to see how it performs and what resources it ends up consuming. Then tweak your dataflows design implementation to achieve better throughput.

Thanks,

Matt

Re: We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

dhieru — Thu, 12 Jul 2018 02:37:00 GMT

make sure that you install only Nifi on these nodes. Do not club Kafka and zookeeper on nifi nodes. This will imporve network i/o performance.

Dhieru