Nifi out of memory error java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123

[#1]

My Nifi instance of 5 node is not starting with the below error , please help on resolving this , I have tried deleting the repos and setting nifi.flowcontroller.autoResumeState=false in nifi.properties , start nifi service then also getting the same error , please help on this

I have also increased the heap size that also did not resolve this

2021-11-24 14:20:43,413 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: null
at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at org.apache.nifi.util.file.FileUtils.copy(FileUtils.java:304)
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:110)
at org.apache.nifi.controller.StandardFlowService.createDataFlow(StandardFlowService.java:608)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:467)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1028)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:301)

[#2]

@Neil_1992 

The stack trace indicated your NiFi is running out of heap memory while loading your flow.xml.gz during startup?

What version of NiFi are you using?  (Have you tried upgrading?)
What version of Java is being used? (Have you tried upgrading?)

Does your flow.xml.gz contain a lot of NiFi templates?
Have you tried increasing the xms and xmx setting in the NiFi bootstrap.conf file?

I suspect if you renamed your flow.xml.gz file that your NiFi would startup just fine.  Since i am guessing you do not want to lose your flow, I would increase the heap memory allocation for your NiFi.  Once up, I would look for way to reduce the heap usage of your dataflow(s) by removing old unused dataflows, and downloading templates so they are preserved outside of NiFi, and then delete templates inside of NiFi.

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

[#3]

Hi Matt,

I found the actual Issue , Though the flow.xml.gz file is around 50 mb but when I unzip it , Its becoming 3.17 GB , as Nifi operates under one jvm it is not able to handle it and throwing Bytearray output stream  huge capacity error . I have increased Heap to 200GB for both XMS and XMX but it is not able to handle the Flow.xml.gz file and not able to start Nifi . I am trying to break the file into 3 chunks and retrieve the flows that way 

[#4]

I am able to see some anomalies in the flow.xml.gz file when i unzipped and looking around it , one of the property value inside of it is having almost a million null values , this is must be some kind of bug and resulting in this bytearray outofmemory error , take a look at the below snap

[#5 · Accepted solution]

@Neil_1992 

I strongly recommend not setting your NiFi heap to 200GB.
Java reserves the XMS space and grows to the XMX space as space is requested.
Java Garbage Collection (GC) execution to reclaim heap no longer being used does not kick in to ~80% of heap is used. That means GC in your case would kick in at around 160+ GB of used heap.  All GC execution is stop-the-world activity which means your NiFi will stop doing anything until GC completes. This can lead to long pauses resulting node disconnections, issues with timeouts with dataflows to external services ,etc.

When it comes to flow.xml.gz file, you are correct that it is uncompressed and loaded into heap memory.
The flow.xml.gz contains:

• Everything you add via the NiFi UI to the canvas (processors, RPG, input/output ports, funnels. labels, PGs, controller services, reporting tasks, connections, etc.).  This includes all the configuration of fro each of those components.
• NiFi Templates are also stored in the flow.xml.gz, uncompressed and loaded in to heap as well.  Once a NiFi template is created, it should be downloaded, stored outside of NiFi, and local copy of template inside of NiFi deleted.  

As far as your specific flow.xml.gz, I see a closing tag "</property>" following that indicates that some component has a property which typically consists of a "name" and "value" with the huge null laced null strings in the value field.  I'd scroll up to see which component this property belongs to and then check why this value was set.  Maybe it was a copy paste issue?  Maybe this is just part of some template that was created with this large string for some purpose?  Nothing here says with any certainty that there is a bug.

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt