Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

Solved Go to solution
Highlighted

We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

New Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

Super Guru
4 REPLIES 4

Re: We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

Super Guru

Re: We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

New Contributor

@sunile.manjee we have configuration like 256GB of RAM,12*2TB HDD,600GB SSD,dual 16 core processor for each node...my question is because of this shall we process throughput of 100000 events/sec and 100MB/s with 3 or 4 nodes ?

Re: We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

Master Guru

@Rakesh S

-

NiFi is very capable of hitting those numbers.... But it is impossible for anyone to say base on the little information provided.
NiFi requires very little resources to run. It is the dataflows/use cases which users design and how they implement those in NiFi that impact load on the available resources of the server.

-

So while those throughput numbers may be possible in one users dataflow implementation, they may not be for another user.

The best option is to perform so load testing in your Final NiFi design to see how it performs and what resources it ends up consuming. Then tweak your dataflows design implementation to achieve better throughput.

-

Thanks,

Matt

Re: We have required throughput of 100000 events/sec and 100MB/s in Nifi....How can we achieve this throughput?...share nifi configuration and nifi clustering

Contributor

make sure that you install only Nifi on these nodes. Do not club Kafka and zookeeper on nifi nodes. This will imporve network i/o performance.

Dhieru

Don't have an account?
Coming from Hortonworks? Activate your account here