Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to debug each nifi processor ?

avatar
Rising Star

How to debug each nifi processor interms of how much RAM, network , how many flowfiles ..etc .

Or is it possible to write seperate log files for each nifi processor ? Please let us know the process . thanks

1 ACCEPTED SOLUTION

avatar
Guru

The first and most straightforward approach is simply to run tail -f | grep <ProcessorName> against log/nifi-app.log and append it to a log. Ex

tail -f log/nifi-app.log | grep ExecuteScript >> ExecuteScript.log

You could even use NiFi itself. Create a flow to tail the log file, extract text that you want (e.g. ExecuteScript) and route this to a file. GetFile (set for tail) -> ExtractText -> PutFile. You could make this as complex as you like (e.g one flow but conditionally route extracted text to each target file based on the processor name).

Finally, you could build scripts around the NiFi Rest API to monitor your flows and processors: https://nifi.apache.org/docs/nifi-docs/rest-api/ E.g. API for Reporting Tasks, Flow, System Diagnostics.

View solution in original post

5 REPLIES 5

avatar
Guru

The first and most straightforward approach is simply to run tail -f | grep <ProcessorName> against log/nifi-app.log and append it to a log. Ex

tail -f log/nifi-app.log | grep ExecuteScript >> ExecuteScript.log

You could even use NiFi itself. Create a flow to tail the log file, extract text that you want (e.g. ExecuteScript) and route this to a file. GetFile (set for tail) -> ExtractText -> PutFile. You could make this as complex as you like (e.g one flow but conditionally route extracted text to each target file based on the processor name).

Finally, you could build scripts around the NiFi Rest API to monitor your flows and processors: https://nifi.apache.org/docs/nifi-docs/rest-api/ E.g. API for Reporting Tasks, Flow, System Diagnostics.

avatar
Expert Contributor

@Greg Keys I would like to apply the same approach, but i prefer to extract logs per process group/flow instead at the processor level. However i see no process group/flow related information in the NiFi logs. Further, i see no processor name as well but only processor type, meaning i have around 20 ExtractText processors with different (customizable) names, it is quite some effort to extract information based on processor IDs only.

avatar
Master Guru

Look at the Ambari Metrics and Grafana reports

http://hortonworks.com/blog/3-things-to-know-hdf-2-0/

avatar
Guru

This post shows how to very quickly build separate logs for each processor (or however else you wish to customize logs)

https://community.hortonworks.com/articles/65027/nifi-easy-custom-logging-of-diverse-sources-in-mer....

avatar
Expert Contributor

@Greg Keys Your blog is great to clarify a bit the approach one can use, however i run into problems with diverse format of logs that span through multiple lines and then with this approach one logline can be split in multiple flowfiles which is not desired.

Is there a way to use SplitText to split the files in specific patterns?