Support Questions
Find answers, ask questions, and share your expertise

How to debug each nifi processor ?

Solved Go to solution

How to debug each nifi processor ?

How to debug each nifi processor interms of how much RAM, network , how many flowfiles ..etc .

Or is it possible to write seperate log files for each nifi processor ? Please let us know the process . thanks

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to debug each nifi processor ?

Guru

The first and most straightforward approach is simply to run tail -f | grep <ProcessorName> against log/nifi-app.log and append it to a log. Ex

tail -f log/nifi-app.log | grep ExecuteScript >> ExecuteScript.log

You could even use NiFi itself. Create a flow to tail the log file, extract text that you want (e.g. ExecuteScript) and route this to a file. GetFile (set for tail) -> ExtractText -> PutFile. You could make this as complex as you like (e.g one flow but conditionally route extracted text to each target file based on the processor name).

Finally, you could build scripts around the NiFi Rest API to monitor your flows and processors: https://nifi.apache.org/docs/nifi-docs/rest-api/ E.g. API for Reporting Tasks, Flow, System Diagnostics.

View solution in original post

5 REPLIES 5
Highlighted

Re: How to debug each nifi processor ?

Guru

The first and most straightforward approach is simply to run tail -f | grep <ProcessorName> against log/nifi-app.log and append it to a log. Ex

tail -f log/nifi-app.log | grep ExecuteScript >> ExecuteScript.log

You could even use NiFi itself. Create a flow to tail the log file, extract text that you want (e.g. ExecuteScript) and route this to a file. GetFile (set for tail) -> ExtractText -> PutFile. You could make this as complex as you like (e.g one flow but conditionally route extracted text to each target file based on the processor name).

Finally, you could build scripts around the NiFi Rest API to monitor your flows and processors: https://nifi.apache.org/docs/nifi-docs/rest-api/ E.g. API for Reporting Tasks, Flow, System Diagnostics.

View solution in original post

Highlighted

Re: How to debug each nifi processor ?

Rising Star

@Greg Keys I would like to apply the same approach, but i prefer to extract logs per process group/flow instead at the processor level. However i see no process group/flow related information in the NiFi logs. Further, i see no processor name as well but only processor type, meaning i have around 20 ExtractText processors with different (customizable) names, it is quite some effort to extract information based on processor IDs only.

Highlighted

Re: How to debug each nifi processor ?

Super Guru

Look at the Ambari Metrics and Grafana reports

http://hortonworks.com/blog/3-things-to-know-hdf-2-0/

Highlighted

Re: How to debug each nifi processor ?

Guru

This post shows how to very quickly build separate logs for each processor (or however else you wish to customize logs)

https://community.hortonworks.com/articles/65027/nifi-easy-custom-logging-of-diverse-sources-in-mer....

Highlighted

Re: How to debug each nifi processor ?

Rising Star

@Greg Keys Your blog is great to clarify a bit the approach one can use, however i run into problems with diverse format of logs that span through multiple lines and then with this approach one logline can be split in multiple flowfiles which is not desired.

Is there a way to use SplitText to split the files in specific patterns?

Don't have an account?