Capture error messages

[#1]

I have a Nifi data flow template and I would like to capture the error messages at each stage of the processors in case if there is an error at any stage of the flow and route them to my control database.The thing is if one of the flowfiles fails, I need someway of knowing that error and getting the error.So I need a way of detecting the failure and then getting that message back to our control database.So that I can see at what stage my job has failed or being successful. Is there a possible way to do this?

TIA

[#2 · Accepted solution]

@Patrick

There may issues external to a processors code execution that can result in a stoppage in a dataflow.

The Monitor activity processor can be placed inline anywhere in your flow to monitor FlowFiles passing through it on a configurable interval. The processor then will generate alert type FlowFiles when a activity is outside of those thresholds. You could then send those generated alert messaged to a putEmail processor to send out a notification of an outage.

Not all failures are unexpected, Often times a second attempt is successful. Setting up a count loop is a good way of having FlowFiles that are routed to failure to to try x number of times in a loop before taking some other action.

https://cwiki.apache.org/confluence/download/attachments/57904847/Retry_Count_Loop.xml?version=1&mod...

You may also want to consider looking at the SiteToSiteBulletinReportingTask. NiFi processor produce bulletins for ERROR and WARN level logs. With this reporting task, you can feed these produced bulletins back to same NiFi or another NiFi where they can be handled like any other FlowFile and pushed to what ever database you want.

Thank you,

Matt

*** If you found this answer addressed your question, please take a moment to login and click "accept"