Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Performance Management in Nifi

Performance Management in Nifi

Hi Team,

I am working on Nifi flows for my team and facing issues on managing the performance of flow while using SplitText/ReplaceText/ExtractText.

One of the workaround i got is to use 3-4 SplitText Processors, but not sure how can we do the same.

Please let me know what changes i required in my flows to ingest and transform 1 GB of data records.

PS. Please refer this link to know all of my use cases -

https://community.hortonworks.com/articles/66861/nifi-etl-removing-columns-filtering-rows-changing.h...

Thanks in advance!

Regards,

Garima.

2 REPLIES 2
Highlighted

Re: Performance Management in Nifi

Guru

Could you post more about the following:

  • cluster size including cores per node
  • performance bottleneck (which processor(s))
  • time to complete processing of 1GB
Highlighted

Re: Performance Management in Nifi

Super Guru

Here are some recommended hardware configs based on throughput & etc. can you verify you adhere (or at least close) to the reocmmendations?

Throughput

Target

Number of NiFi nodes CPU Cores/node Number of disks/node, size of each disk (RAID 5/10) RAM/node Ideal Networking Setup
50 MB/s,

1000 events/s

3 16+ 6+, 1TB 8+ GB 1 Gigabit bonded NICs
100 MB/s,

10,000 events/s

5 16+ 6+, 2TB 8+ GB 1 Gigabit bonded NICs
200 MB/s,

100,000 events/s

7 24+ 12+, 4TB 16+ GB 10 Gigabit bonded NICs
400 MB/s,

100,000+ events/s

9 24+ 12+, 8TB 16+ GB 10 Gigabit bonded NICs
Don't have an account?
Coming from Hortonworks? Activate your account here