Member since
02-23-2018
2
Posts
0
Kudos Received
0
Solutions
02-27-2018
04:36 PM
That was indeed the issue. The Pulsar consumer.receiveAsync() method would return a CompletableFuture object, that would hang around until the broker had issued an acknowledgment. Given the speed of the Pulsar consumer, there were 10s of thousands of these objects (each with an individual FlowFile) in memory at any given time. I add an ExecutorCompletionService to the class, which limits the number of instances of CompletableFuture's at any given time, and was able to run a 24 hour stress test w/o any memory leaks.
... View more
02-23-2018
04:56 PM
Experts, I have written a custom processor and associated unit tests. Everything works well in a limited environment. However, when I set up a simple stress test, the processor causes NiFi itself to crash and restart over and over again. When I ran analysis on the heap dump generated from NiFi, it showed that the leaks suspect was the org.apache.nifi.controller.repository.StandardProcessSession class, which was consuming 81% of the heap. My questions are: 1) What could be causing this behavior 2) Since this class is a core class, how do I go about finding where these instances are being created so I can identify the leak?
... View more
Labels:
- Labels:
-
Apache NiFi