Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Debugging NiFi flow

Solved Go to solution
Highlighted

Debugging NiFi flow

Hi,

I have a NiFi flow that consumes lot of memory. How can I diagnosis the flow to isolate the problem and find the processor that

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Debugging NiFi flow

Guru

You can use NiFis native reporting to monitor:

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks

For your needs, there is a memory monitor which checks the amount of Java Heap available in the JVM for a particular JVM Memory Pool. If the amount of space used exceeds some configurable threshold, will warn (via a log message and System-Level Bulletin) that the memory pool is exceeding this threshold.

See this post for more on NiFi Reporting:

https://community.hortonworks.com/questions/69004/nifi-monitoring-processor-and-nifi-service.html#an...

View solution in original post

3 REPLIES 3
Highlighted

Re: Debugging NiFi flow

Contributor
@Houssam Manik have you enabled debug logging for the processors that you are using to try and investigate further? Your configuration can be found at <NIFI_HOME>/conf/logback.xml, you can set processors etc to debug level to investigate further, eg:
<logger name="org.apache.nifi.processors.standard.GetFile" level="DEBUG"/>

Are you running with a default configuration? If so you could try bumping your memory settings in conf/bootstrap.conf. They default to 512mb, you may want to try bumping it up to 1024mb.

If you could post more about your flow/use case I'm sure we can help narrow down which processor is causing the problem.

Highlighted

Re: Debugging NiFi flow

Guru

You can use NiFis native reporting to monitor:

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks

For your needs, there is a memory monitor which checks the amount of Java Heap available in the JVM for a particular JVM Memory Pool. If the amount of space used exceeds some configurable threshold, will warn (via a log message and System-Level Bulletin) that the memory pool is exceeding this threshold.

See this post for more on NiFi Reporting:

https://community.hortonworks.com/questions/69004/nifi-monitoring-processor-and-nifi-service.html#an...

View solution in original post

Highlighted

Re: Debugging NiFi flow

Cloudera Employee

In Apache Nifi 1.1 there is also the new back pressure indicator bars that you can configure. These may be able to give you a clue as to where the bottleneck is...

Don't have an account?
Coming from Hortonworks? Activate your account here