Skip to main content
sarithe
sarithe
Silver ⭐️⭐️
July 5, 2023
Question

Nifi Perfromnce issues

Hi Team,

 

We are facing Nifi performance issues form a while as we increased no of processors and processor groups. We have increased thread count but still facing the issue. CPU utilization is also not reaching minimum level it is utilizing less than 50%. How to increase the performance in Nifi. We have been using 13.2 version till now and migrated to 18.0 to see if there is performance improvement but still facing the same. Could anyone suggest how to improve performance.

9 replies

MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
July 5, 2023

@sarithe 

The NiFi components (Processors, input ports. output ports, funnels, etc) will request a thread from the configured "Max timer Driven thread" pool in NiFi each time the processor is scheduled to execute.  Adding more components will NOT increase the size of the thread pool (which is only set to 10 default out of the box).  Generally speaking, The recommended starting size for your "Max Timer Driven Thread" should be 2 - 4 times the number of cores you have on yoru NiFi server (Example - 16 cores would have thread pool set between 32 and 64).  After adjusting your thread pool, you should monitor CPU utilization on your server while your dataflows are being executed within NiFi.  You may find that your NiFi components execute very fast threads (milliseconds) and cpu load average remains low meaning you could set the thread pool even larger.  Or you find your core load average remains very high meaning you should reduce your thread pool size.   

The configuration for the "Max Timer Driven Thread" can be found from NiFi UI --> Global menu (upper right corner) --> Controller Settings

Note: There is also an "Event Driven Thread Pool" which you should NOT edit.  It is used by processors that have been configured to use the "Event Driven" scheduling strategy (No processors use this by default).  This strategy was experimental and has been deprecated. It willl be removed completely in next major Apache NiFi release.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Translate
sarithe
saritheAuthor
Silver ⭐️⭐️
July 6, 2023

Hi Matt,

 

Thanks for sharing the insights. We are using 3 node cluster and each cluster having 16cores and we have set timer driven thread count to 160. And seeing core utilization as 8%. when we are getting huge data like 2 GB in one flow, the ingestion is becoming dead slow though increase in threads.

Translate
sarithe
saritheAuthor
Silver ⭐️⭐️
July 11, 2023

Yes, whenever we are seeing slowness increasing the concurrent tasks to 6 to 7 but still I am not seeing that much difference, at some time it will be better when increased concurrent tasks. This behaviour is mainly seen for Replace Text processor.

Translate
joseomjr
joseomjr
Gold ⭐️⭐️
July 30, 2023

NiFi is awesome with so many out-of-the-box processors...however, I have found sometimes a very specialized scripted Groovy processor that fetches 1000 or more files at a time to perform significantly faster... especially if your custom processor consolidates several processors into 1. 

Translate
Terms & ConditionsAccessibility statement