Nifi cluster CPU Utilization / RAM always peaking .

[#1]

Hi Team , 

I need some advice on how to tune our Nifi Cluster. We have a 3 Node Nifi Cluster that processes events in a real time streaming fashion which means our flows are running 24 by 7 . Each of the Nifi Linux servers have a RAM of 31GB. We have set the JVM memory setting in the bootstrap conf file to 28 G. 

#JVM memory Settings

java.arg.2=-Xms28G

java.arg.3=-Xms28G

However when I run a free -h command on the servers this is what I get which clearly shows that the used Memory is 30 G. 

As a result of this , we keep getting incidents raised from the Linux team stating that the Nifi servers are always hovering over the  95 % mark and on top of that if any processor in the flow starts misbehaving , it shoots up to 100 percent and the server goes unresponsive. 

So my question is , even after setting the threshold to 28 G in the bootstrap conf file , why do we still see Nifi taking up almost all the memory allocated to the server ? 

@Matt 

[#2 · Accepted solution]

in your nifi app log if you have "out of memory" errors it may indicate that you need to add more memory (or a possible memory leak)

increasing heap too much will cause long GC (especially with Old Generation Objects)

you may need to use a more efficient GC if you use large heap size

in bootstrap.conf uncomment line : #java.arg.13=-XX:+UseG1GC

as you know, some processors are memory intensive

queues between processors use memory, ... until nifi.queue.swap.threshold is reached (default 20000 -> if reached nifi will push to disk remaining DFs  ) -> too much swapping in queues will realy affect performances. (memory + disk)

there are many points that can help you to determine where the PB may be

for ex. (processor 1) --> queue --> (processor 2)

- check queue occupation : if always full for some processors parallelize more in following processors (especially if processor a re not memory intensive, and for ex if the processor is cpu intenssive and look likes it's not ingestings quickly from queue adding more threads will help to relieve the queue ). also ading back-pressus/control-rate before. 

in tha case just checking "in" and "out" of processor 2 will confirm that  (high IN and low OUT), increase nb of threads -> (threadsScheduling>Concurrent Tasks), sometimes tuning "Run Schedule" and "Run Duration" may also help depending of ingres DFs nature/size and processor type and how parallelization is impacted

take a look at "Total queued data" in the status bar (under components bar on top of the UI)

keep in mind that processors may be I/O or CPU or memory intensive or all, and parallelizing more to reduce backlogged data may solve memory PBs but it will cause more threads overuse.

adding more NIFI nodes will remain the ultimate solution to increase resources

to increase performances, you may also want to take a look at

nifi.provenance.repository.index.threads

nifi.provenance.repository.query.threads

nifi.provenance.repository.index.shard.size

using different filesystems (with good IOPS) for directories for provenance repository, content repository, flowfile repository

and more 

[#3 · Accepted solution]

in your nifi app log if you have "out of memory" errors it may indicate that you need to add more memory (or a possible memory leak)

increasing heap too much will cause long GC (especially with Old Generation Objects)

you may need to use a more efficient GC if you use large heap size

in bootstrap.conf uncomment line : #java.arg.13=-XX:+UseG1GC

as you know, some processors are memory intensive

queues between processors use memory, ... until nifi.queue.swap.threshold is reached (default 20000 -> if reached nifi will push to disk remaining DFs  ) -> too much swapping in queues will realy affect performances. (memory + disk)

there are many points that can help you to determine where the PB may be

for ex. (processor 1) --> queue --> (processor 2)

- check queue occupation : if always full for some processors parallelize more in following processors (especially if processor a re not memory intensive, and for ex if the processor is cpu intenssive and look likes it's not ingestings quickly from queue adding more threads will help to relieve the queue ). also ading back-pressus/control-rate before. 

in tha case just checking "in" and "out" of processor 2 will confirm that  (high IN and low OUT), increase nb of threads -> (threadsScheduling>Concurrent Tasks), sometimes tuning "Run Schedule" and "Run Duration" may also help depending of ingres DFs nature/size and processor type and how parallelization is impacted

take a look at "Total queued data" in the status bar (under components bar on top of the UI)

keep in mind that processors may be I/O or CPU or memory intensive or all, and parallelizing more to reduce backlogged data may solve memory PBs but it will cause more threads overuse.

adding more NIFI nodes will remain the ultimate solution to increase resources

to increase performances, you may also want to take a look at

nifi.provenance.repository.index.threads

nifi.provenance.repository.query.threads

nifi.provenance.repository.index.shard.size

using different filesystems (with good IOPS) for directories for provenance repository, content repository, flowfile repository

and more 

[#4 · Accepted solution]

Thanks , enabling the GIGC has helped along with reducing the Heap JVM  settings in the bootstrap conf file. Thanks again