How to debug each nifi processor ?

[#1]

How to debug each nifi processor interms of how much RAM, network , how many flowfiles ..etc .

Or is it possible to write seperate log files for each nifi processor ? Please let us know the process . thanks

[#2 · Accepted solution]

The first and most straightforward approach is simply to run tail -f | grep <ProcessorName> against log/nifi-app.log and append it to a log. Ex

tail -f log/nifi-app.log | grep ExecuteScript >> ExecuteScript.log

You could even use NiFi itself. Create a flow to tail the log file, extract text that you want (e.g. ExecuteScript) and route this to a file. GetFile (set for tail) -> ExtractText -> PutFile. You could make this as complex as you like (e.g one flow but conditionally route extracted text to each target file based on the processor name).

Finally, you could build scripts around the NiFi Rest API to monitor your flows and processors: https://nifi.apache.org/docs/nifi-docs/rest-api/ E.g. API for Reporting Tasks, Flow, System Diagnostics.

[#3]

@Greg Keys I would like to apply the same approach, but i prefer to extract logs per process group/flow instead at the processor level. However i see no process group/flow related information in the NiFi logs. Further, i see no processor name as well but only processor type, meaning i have around 20 ExtractText processors with different (customizable) names, it is quite some effort to extract information based on processor IDs only.

[#4]

Look at the Ambari Metrics and Grafana reports

http://hortonworks.com/blog/3-things-to-know-hdf-2-0/

[#5]

This post shows how to very quickly build separate logs for each processor (or however else you wish to customize logs)

https://community.hortonworks.com/articles/65027/nifi-easy-custom-logging-of-diverse-sources-in-mer....

[#6]

@Greg Keys Your blog is great to clarify a bit the approach one can use, however i run into problems with diverse format of logs that span through multiple lines and then with this approach one logline can be split in multiple flowfiles which is not desired.

Is there a way to use SplitText to split the files in specific patterns?