NIfi ram consumption

[#1]

Hi all,

We have a Nifi community version and we noted that the service is using 46GB of ram on a virtual server that has 47GB.

We used these settings to limit the amount of ram to use
Java. Ag. 2=-Xms21g
Java. Ag. 3=-Xmx42g
But the issue remains.

Can you suggest how to address this issue?

Regards,
Fortunato

[#2]

@2ir Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @mburgess  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

@2ir 
NiFi can consume heap (native) and non-heap (non native) memory. This commonly happens with processor that create jetty servers for listening for inbound requests, scripting processors that execute child scripts, processors that execute OS commands,  NiFi bootstrap parent process,  etc...  So XMX heap memory is only part of the memory that can be used by a Java application.  From within the NiFi UI --> global menu --> cluster or summary, you can see that actual amount of heap utilized using (JVM tab cluster UI or summary system diagnostics UI).

I would advise against setting your Heap so high when you have 47 GB total memory.  It is likely that your OS if linux based is going to invoke OOM killer to kill the NiFi process to protect the OS.  I'd advise reducing your heap usage to XMS and XMX of 24 GB.

Re-evaluating your dataflows for high memory use processor and making sure they are optimally configured is next steps.  The embedded documentation for the the processor components will each have a "System Resource Considerations" section that will tell you if the processor has the potential to use high memory or high cpu.  For processors with potential for high heap usage, be careful with concurrent task configuration. Default on current tasks is always 1.  Increasing it is like adding a second copy of the processor allowing multiple concurrent executions thus increasing the heap usage. (Example: ReplaceText 1.25.0)

You'll want to be careful using templates (deprecated) as any templates generated and held in NiFI consume heap.  FlowFile metadata is held in heap so avoid creating FlowFile with large attributes (like extracting content to attributes). Use Record base processor whenever possible to reduce numbers of individual FlowFiles.  Use a NiFi cluster instead of standalone NiFi to spread FlowFile load across multiple NiFi instances.

Monitor heap usage and and collect heap dumps to analyze what is consuming the heap.

Hope this helps you with your investigative journey.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#4]

Hello Mat,

thank you for your support. We have a 3 node cluster (NiFi version is 1.19.0) and the scenario I'm presenting is the one with a unique execution of a single flow, scheduled to run once a day. The flow is an ETL type (no http jetty instance is invoked). The issue is that the ram occupancy, not increment at the same time for each run and remains high even if the flow is not running.

We have tested the flow for one month. The ram occupancy was stable for 3 weeks and in the last one we had the abnormal consumption of the java process invoked by nifi service. Since the last week, the ram occupancy was not decreased.

Thank you,

Fortunato

[#5]

@2ir 

Heap usage continues to grow until utilization exceeds ~80% at which time heap Garbage Collection (GC) is executed to release unused heap memory allocations.   Even if there are no FlowFiles or running flows in NiFi, if the GC threshold has not been reached, the heap would not be cleaned out.   First question is why is our NiFi heap set so high already?  Where you encountering OutOfMemory (OOM) issues with smaller heap settings.  You'll get the best performance using the smallest workable heap values.  Typical NiFi heap size recommendations in most scenarios is XMS and XMX of 16 GB or less.

You stated that consumption was stable for 3 weeks and then changed:

• What changed about the content being ingested and created (size, number, etc)?
• Where any new dataflows added or recently enabled?
• What processor types are being used in your dataflows?
• Are you leaving any FlowFiles queued up in dataflow connections?
• Are you writing large attributes to FlowFiles?
• Are you extracting content from FlowFiles into attributes?

Aside from that you would need to look at hep dumps to see what is consuming the majority of the space to maybe identify the components responsible.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#6]

Hello Matt,

Nothing was changed in the last month: the environment we are using for the memory issue is isolated and not used. At this moment only one flow is scheduled. I can enable the java arg 13 that is disabled.

#java.arg.13=-XX:+UseG1GC

Can this issue be related to the open version of java?

openjdk version "11.0.20" 2023-07-18 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.20.0.8-1.el7_9) (build 11.0.20+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.20.0.8-1.el7_9) (build 11.0.20+8-LTS, mixed mode, sharing)

Thank you,

Fortunato

[#7 · Accepted solution]

@2ir 

I doubt it is related to Java version being used.  Of course we would always recommend using the latest update version for a NiFi supported Java version.


As far as using G1GC, it is commented out because Java 8 has many issues when using G1GC and the Java community decided to address those bugs and improvements in Java 9+ versions.  Since you are using Java 11, G1GC would be a better option.  With that lin commented out, NiFi does not specify and GC for yoru Java and whatever the Default GC defined within your java release would be used.  That line allows you to override yoru java default and specify the GC you want to use.

Memory issues are often attributed to issues in custom components added to the NiFi deployment or dataflow design choices. Hence all the dataflow related input i provided previously.  You never mentioned if you were encountering any out of memory (OOM) error logs in your NiFi logs?   If not, do you see any OOMs if you decrease your Heap memory setting which you have set rather high already.  Also recommend setting both xms and xmx to same value.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#8]

Hi Matt,

we configured 24GB as xms and xmx parameters. For now we have a normal use of memory without OOM errors.

Thank you,

Fortunato

[#9]

@2ir Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.