Debugging NiFi flow

[#1]

Hi,

I have a NiFi flow that consumes lot of memory. How can I diagnosis the flow to isolate the problem and find the processor that

[#2]

@Houssam Manik have you enabled debug logging for the processors that you are using to try and investigate further? Your configuration can be found at <NIFI_HOME>/conf/logback.xml, you can set processors etc to debug level to investigate further, eg:

<logger name="org.apache.nifi.processors.standard.GetFile" level="DEBUG"/>

Are you running with a default configuration? If so you could try bumping your memory settings in conf/bootstrap.conf. They default to 512mb, you may want to try bumping it up to 1024mb.

If you could post more about your flow/use case I'm sure we can help narrow down which processor is causing the problem.

[#3 · Accepted solution]

You can use NiFis native reporting to monitor:

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks

For your needs, there is a memory monitor which checks the amount of Java Heap available in the JVM for a particular JVM Memory Pool. If the amount of space used exceeds some configurable threshold, will warn (via a log message and System-Level Bulletin) that the memory pool is exceeding this threshold.

See this post for more on NiFi Reporting:

https://community.hortonworks.com/questions/69004/nifi-monitoring-processor-and-nifi-service.html#an...

[#4]

In Apache Nifi 1.1 there is also the new back pressure indicator bars that you can configure. These may be able to give you a clue as to where the bottleneck is...