question Re: How to identify source processor for failed NiFi flowfiles written to a log in Archives of Support Questions (Read Only)

How to identify source processor for failed NiFi flowfiles written to a log

Raj_B — Tue, 24 Jan 2017 06:02:13 GMT

Hello,

We're logging failed flowfiles to local file system; when reviewing the failed flowfiles, there is no information to tell us at which processor a flowfile failed; I was thinking about tagging the flowfiles at each processor, with the processor name it's about to enter; and when logging the failed flowfile, embed this tag/attribute into the file name; so looking at the file name you could tell at what processor the flowfile failed; I'm sure this is not the best way, so any suggestions on how to do this better.

Thanks.

Re: How to identify source processor for failed NiFi flowfiles written to a log

rbalakrishnantc — Wed, 25 Jan 2017 15:46:46 GMT

@Raj B

You can do this in many ways,

1. You can have an update attribute processor at the beginning of every Processor Group to specify the module name. But this cannot be replicated at processor level. Here, you keep updating the same attribute throughout the flow

2. You have have a 'Metadata' -> as a table or a flat file that will log all the attributes of the flow file. Please check AttributeToJson processor. This can log flow attributes both as an attribute or content. Also, note that every nifi processor adds additional 'error' attribute in case of any error which can be captured by AttributeToJson processor.

Hope this helps

Re: How to identify source processor for failed NiFi flowfiles written to a log

Raj_B — Thu, 26 Jan 2017 21:53:52 GMT

Thanks @Balakrishnan Ramasamy; didn't know about the 'error' attribute on failed flowfiles.

Re: How to identify source processor for failed NiFi flowfiles written to a log

Raj_B — Mon, 19 Aug 2019 08:07:06 GMT

Hi @Balakrishnan Ramasamy, do you know where I can find documentation about the 'error' attribute; I was looking in the NiFi documentation and it does not show the 'error' as an available attribute. Thanks.

Re: How to identify source processor for failed NiFi flowfiles written to a log

rbalakrishnantc — Mon, 19 Aug 2019 08:06:58 GMT

@Raj B

Please check for the 'Writes Attribute' in 'usage' of the processor. It describes the attribute each processor writes in case of failure. Below is the Usage of 'ExecuteStreamCommand' and it details 4 different error it writes during the execution.

Re: How to identify source processor for failed NiFi flowfiles written to a log

MattWho — Tue, 31 Jan 2017 22:33:53 GMT

@Raj B

Not all NiFi Processors will write attributes to FlowFiles about failures or errors. The documentation for each processor should include what attributes are written by that processor and what information those attributes will contain. There is no global enforcement by the NiFi controller on what attributes a processor must create. This is completely in the control of the developer who wrote each processor.

That being said, it is good practice that any processor that has a "Failure" relationship should output an "Error" level log message that dictates the nature of the failure. This Error log message would contain the specific processor that produced the ERROR as well as information on the specific FlowFile that was routed to failure and the nature of the failure.

It is possible to build a dataflow that monitors NiFi's nifi-app.log (TailFile processor) for ERROR log messages, parses out the relevant information and pass that along to some monitoring system.

Thanks,

Matt

Re: How to identify source processor for failed NiFi flowfiles written to a log

Raj_B — Wed, 01 Feb 2017 21:48:54 GMT

Thank you, I never paid attention to the "Writes Attributes" and didn't know what they were for.

Re: How to identify source processor for failed NiFi flowfiles written to a log

Raj_B — Wed, 01 Feb 2017 21:52:44 GMT

Thanks @Matt, parsing the nifi-app.log sounds like a better approach to capture errors at each processor and be able to know where the flowfile failed. Perhaps Splunk might be a good fit for this task.

Re: How to identify source processor for failed NiFi flowfiles written to a log

ektapanjwani20 — Wed, 04 Sep 2019 12:02:19 GMT

writes attribute of some processors (SplitRecord, ReplaceText ) does not contains the error it writes during execution.

How and where to identify the error in that case?