Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi out of memory error java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123

avatar
Explorer

My Nifi instance of 5 node is not starting with the below error , please help on resolving this , I have tried deleting the repos and setting nifi.flowcontroller.autoResumeState=false in nifi.properties , start nifi service then also getting the same error , please help on this

 

I have also increased the heap size that also did not resolve this

 

2021-11-24 14:20:43,413 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: null
at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at org.apache.nifi.util.file.FileUtils.copy(FileUtils.java:304)
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:110)
at org.apache.nifi.controller.StandardFlowService.createDataFlow(StandardFlowService.java:608)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:467)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1028)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:301)

1 ACCEPTED SOLUTION

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Super Mentor

@Neil_1992 

 

The stack trace indicated your NiFi is running out of heap memory while loading your flow.xml.gz during startup?

What version of NiFi are you using?  (Have you tried upgrading?)
What version of Java is being used? (Have you tried upgrading?)

Does your flow.xml.gz contain a lot of NiFi templates?
Have you tried increasing the xms and xmx setting in the NiFi bootstrap.conf file?


I suspect if you renamed your flow.xml.gz file that your NiFi would startup just fine.  Since i am guessing you do not want to lose your flow, I would increase the heap memory allocation for your NiFi.  Once up, I would look for way to reduce the heap usage of your dataflow(s) by removing old unused dataflows, and downloading templates so they are preserved outside of NiFi, and then delete templates inside of NiFi.

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.


Thank you,

Matt

avatar
Explorer

Hi Matt,

 

I found the actual Issue , Though the flow.xml.gz file is around 50 mb but when I unzip it , Its becoming 3.17 GB , as Nifi operates under one jvm it is not able to handle it and throwing Bytearray output stream  huge capacity error . I have increased Heap to 200GB for both XMS and XMX but it is not able to handle the Flow.xml.gz file and not able to start Nifi . I am trying to break the file into 3 chunks and retrieve the flows that way 

avatar
Explorer

I am able to see some anomalies in the flow.xml.gz file when i unzipped and looking around it , one of the property value inside of it is having almost a million null values , this is must be some kind of bug and resulting in this bytearray outofmemory error , take a look at the below snap

 

Neil_1992_0-1641897268878.png

 

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login