Support Questions
Find answers, ask questions, and share your expertise

Debugging NiFi flow

Hi,

I have a NiFi flow that consumes lot of memory. How can I diagnosis the flow to isolate the problem and find the processor that

1 ACCEPTED SOLUTION

Accepted Solutions

Guru

You can use NiFis native reporting to monitor:

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks

For your needs, there is a memory monitor which checks the amount of Java Heap available in the JVM for a particular JVM Memory Pool. If the amount of space used exceeds some configurable threshold, will warn (via a log message and System-Level Bulletin) that the memory pool is exceeding this threshold.

See this post for more on NiFi Reporting:

https://community.hortonworks.com/questions/69004/nifi-monitoring-processor-and-nifi-service.html#an...

View solution in original post

3 REPLIES 3

Contributor
@Houssam Manik have you enabled debug logging for the processors that you are using to try and investigate further? Your configuration can be found at <NIFI_HOME>/conf/logback.xml, you can set processors etc to debug level to investigate further, eg:
<logger name="org.apache.nifi.processors.standard.GetFile" level="DEBUG"/>

Are you running with a default configuration? If so you could try bumping your memory settings in conf/bootstrap.conf. They default to 512mb, you may want to try bumping it up to 1024mb.

If you could post more about your flow/use case I'm sure we can help narrow down which processor is causing the problem.

Guru

You can use NiFis native reporting to monitor:

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks

For your needs, there is a memory monitor which checks the amount of Java Heap available in the JVM for a particular JVM Memory Pool. If the amount of space used exceeds some configurable threshold, will warn (via a log message and System-Level Bulletin) that the memory pool is exceeding this threshold.

See this post for more on NiFi Reporting:

https://community.hortonworks.com/questions/69004/nifi-monitoring-processor-and-nifi-service.html#an...

View solution in original post

Cloudera Employee

In Apache Nifi 1.1 there is also the new back pressure indicator bars that you can configure. These may be able to give you a clue as to where the bottleneck is...