Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi OutOfMemoryError : Java heap space with a specific processor | ConvertJSONToSQL

avatar
Contributor

I am facing a memory issue with a specific processor ConvertJSONToSQL

Not sure if there is a way to configure processor specific memory, as when this error comes i found other processor and services running fine in nifi.

Is there a way to address this apart from looking extending heapspace in general to whole nifi , currently its 8GB.

Below is the stack trace of error:

2018-02-09 05:42:53,197 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.ConvertJSONToSQL ConvertJSONToSQL[id=6f1f8d3a-20b3-3fc5-884e-73f1e7443b58] ConvertJSONToSQL[id=6f1f8d3a-20b3-3fc5-884e-73f1e7443b58] failed to process session due to java.lang.OutOfMemoryError: Java heap space: {} java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3236) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) at java.io.DataOutputStream.writeLong(DataOutputStream.java:224) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:117) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:69) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:56) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:47) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:147) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:96) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:56) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:47) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:143) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:69)

Thanks,

Tarun Kumar

6 REPLIES 6

avatar
Expert Contributor

You should modify bootstrap.conf file. Increase moemory size of java.arg.2 ve java.arg.3 like:

java.arg.2=-Xms512m
java.arg.3=-Xmx512m
to
java.arg.2=-Xms1024m
java.arg.3=-Xmx1024m

avatar
Contributor

Still i am getting same error after 1024m, When i am converting CSV to JSON

avatar
Contributor

Solved for me when i change to 4096m

avatar
Expert Contributor

I am glad it is solved @shreyanshu pare

avatar

@Tarun Kumar @shreyanshu pare

Increasing the Heap size will solve your interim problem. However, I will highly recommend using the best practices for setting up high performing NiFi workflow, mentioned here.

Depending on your operations, you find bumped up heap size insufficient in future. So a couple of reasons, well documented by @Matt Clarke, as to why you may see a very high heap usage.

Every FlowFile that exists consists of two parts, FlowFile content and FlowFile Attributes. While the FlowFile's content lives on disk in the content repository, NiFi holds the "majority" of the FlowFile attribute data in the configured JVM heap memory space. I say "majority" because NiFi does swapping of Attributes to disk on any queue that contains over 20,000 FlowFiles (default, but can be changed in the nifi.properties).

So some common reason for running out of heap memory include:

1. High volume dataflow with lots of FlowFiles active any any given time across your dataflow. (Increase configured nifi heap size in bootstrap.conf to resolve)

2. Creating a large number of Attributes on every FlowFile. More Attributes equals more heap usage per FlowFile. (Increase configured nifi heap size in bootstrap.conf to resolve and/or reduce the configured swap threshold)

3. Writing large values to FlowFile Attributes. Extracting large amounts of content and writing it to an attribute on a FlowFile will result in high heap usage. Try to avoid creating large attributes when possible. (Increase configured nifi heap size in bootstrap.conf to resolve and/or reduce the configured swap threshold)

4. Using the MergeContent processor to merge a very large number of FlowFiles. NiFi can not merge FlowFiles that are swapped, so all these FlowFile's attributes must be in heap when the merge occurs. If merging a very large number of FlowFiles is needed, try using two MergeContent processors in series with one another. Have first merge a max of 10,000 FlowFiles and the second then merge those 20,000 FlowFile files in to even larger bundles. (Increase configured nifi heap size in bootstrap.conf also help)

5. Using the SplitText processor to split one File in to a very large number of FlowFiles. Swapping of a large connection queue will not occur until after the queue has exceeded swapping threshold. The SplitTEXT processor will create all the split FiLowFiles before committing them to the success relationship. Most commonly seen when SpitText is used to split a large incoming FlowFile by every line. It is possible to run out of heap memory before all the splits can be created. Try using two SplitText processors in series. Have the first split the incoming FlowFiles in to large chunks and the second split them down even further. (Increase configured nifi heap size in bootstrap.conf also help)

Hope that helps!

avatar
Master Guru

I like at least 64-128GB of RAM