question How to debug each nifi processor ? in Archives of Support Questions (Read Only)

How to debug each nifi processor ?

srinivaspadala_ — Tue, 25 Oct 2016 10:04:30 GMT

How to debug each nifi processor interms of how much RAM, network , how many flowfiles ..etc .

Or is it possible to write seperate log files for each nifi processor ? Please let us know the process . thanks

Re: How to debug each nifi processor ?

gkeys — Tue, 25 Oct 2016 20:23:47 GMT

The first and most straightforward approach is simply to run tail -f | grep <ProcessorName> against log/nifi-app.log and append it to a log. Ex

tail -f log/nifi-app.log | grep ExecuteScript >> ExecuteScript.log

You could even use NiFi itself. Create a flow to tail the log file, extract text that you want (e.g. ExecuteScript) and route this to a file. GetFile (set for tail) -> ExtractText -> PutFile. You could make this as complex as you like (e.g one flow but conditionally route extracted text to each target file based on the processor name).

Finally, you could build scripts around the NiFi Rest API to monitor your flows and processors: https://nifi.apache.org/docs/nifi-docs/rest-api/ E.g. API for Reporting Tasks, Flow, System Diagnostics.

Re: How to debug each nifi processor ?

TimothySpann — Tue, 25 Oct 2016 20:46:26 GMT

Look at the Ambari Metrics and Grafana reports

http://hortonworks.com/blog/3-things-to-know-hdf-2-0/

Re: How to debug each nifi processor ?

gkeys — Wed, 09 Nov 2016 22:25:42 GMT

This post shows how to very quickly build separate logs for each processor (or however else you wish to customize logs)

https://community.hortonworks.com/articles/65027/nifi-easy-custom-logging-of-diverse-sources-in-mer.html

Re: How to debug each nifi processor ?

foivos — Thu, 04 Jan 2018 23:14:08 GMT

@Greg Keys I would like to apply the same approach, but i prefer to extract logs per process group/flow instead at the processor level. However i see no process group/flow related information in the NiFi logs. Further, i see no processor name as well but only processor type, meaning i have around 20 ExtractText processors with different (customizable) names, it is quite some effort to extract information based on processor IDs only.

Re: How to debug each nifi processor ?

foivos — Wed, 10 Jan 2018 22:45:26 GMT

@Greg Keys Your blog is great to clarify a bit the approach one can use, however i run into problems with diverse format of logs that span through multiple lines and then with this approach one logline can be split in multiple flowfiles which is not desired.

Is there a way to use SplitText to split the files in specific patterns?