[MiNiFi] Memory leak

[#1]

We're running v0.2.1 of the Java version of MiNiFi. Once the instance is running, we periodically send data to it using curl at intervals of ~0.1 seconds. We are using the ListenHTTP processor to accept incoming connections. Memory usage for the application begins to grow slowly as soon as we begin sending requests and doesn't seem to stop. If we use the GenerateFlowFile processor to generate similar data at a similar rate then we don't see the memory grow in the same way.

I tried to use the HandleHttpRequest/Response processors to establish if the leak is caused by the processor or the application. Unfortunately I was unable to get the processors running due to the issues outlined in this question.

For the purposes of testing I have removed all processors apart from the ListenHTTP processor which is auto-terminated so there is just one processor and no connections. I can provide the config.yml if anybody wants to run it locally or, alternatively, I have put a version of the project on GitLab which uses Docker. You can see in the image I have attached that the memory consumption appears to grow steadily.

Any help on resolving this issue would be greatly appreciated.

[#2]

Have you tried taking a heap dump of MiNiFi while this is occurring?

jmap -dump:format=b,file=minifi.bin <pid>

You can use a tool like Eclipse memory analyzer to view the heap dump and try to see what object has the most references in the heap: http://www.eclipse.org/mat/

[#3]

Thanks for the information @Bryan Bende, the memory leak is quite slow so I'm going to leave the application running over the weekend and take a heap dump on Monday and I'll post the result here.

[#4]

@Bryan Bende I had some issues with my machine over the weekend so I was only able to extract a heap dump this morning. You can find it on my GitLab account. The total heap size is just 28.7MB but the overall usage was far higher than that and had grown ~80MB in the first 45 minutes. I opened the file with Eclipse's MemoryAnalyzer tool as you suggested and ran the Leak Suspects report which reported that one instance of "org.apache.nifi.controller.repository.WriteAheadFlowFileRepository" occupies 21.74% of the heap, and one instance of "org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository" occupies 10.65% of the heap. Do you know if this level of memory consumption is expected?