How to identify source processor for failed NiFi flowfiles written to a log

[#1]

Hello,

We're logging failed flowfiles to local file system; when reviewing the failed flowfiles, there is no information to tell us at which processor a flowfile failed; I was thinking about tagging the flowfiles at each processor, with the processor name it's about to enter; and when logging the failed flowfile, embed this tag/attribute into the file name; so looking at the file name you could tell at what processor the flowfile failed; I'm sure this is not the best way, so any suggestions on how to do this better.

Thanks.

[#2]

@Raj B

You can do this in many ways,

1. You can have an update attribute processor at the beginning of every Processor Group to specify the module name. But this cannot be replicated at processor level. Here, you keep updating the same attribute throughout the flow

2. You have have a 'Metadata' -> as a table or a flat file that will log all the attributes of the flow file. Please check AttributeToJson processor. This can log flow attributes both as an attribute or content. Also, note that every nifi processor adds additional 'error' attribute in case of any error which can be captured by AttributeToJson processor.

Hope this helps

[#3]

Thanks @Balakrishnan Ramasamy; didn't know about the 'error' attribute on failed flowfiles.

[#4]

Hi @Balakrishnan Ramasamy, do you know where I can find documentation about the 'error' attribute; I was looking in the NiFi documentation and it does not show the 'error' as an available attribute. Thanks.

[#5]

@Raj B

Please check for the 'Writes Attribute' in 'usage' of the processor. It describes the attribute each processor writes in case of failure. Below is the Usage of 'ExecuteStreamCommand' and it details 4 different error it writes during the execution.

[#6]

Thank you, I never paid attention to the "Writes Attributes" and didn't know what they were for.

[#7]

writes attribute of some processors (SplitRecord, ReplaceText ) does not contains the error it writes during execution. 

How and where to identify the error in that case? 

[#8 · Accepted solution]

@Raj B

Not all NiFi Processors will write attributes to FlowFiles about failures or errors. The documentation for each processor should include what attributes are written by that processor and what information those attributes will contain. There is no global enforcement by the NiFi controller on what attributes a processor must create. This is completely in the control of the developer who wrote each processor.

That being said, it is good practice that any processor that has a "Failure" relationship should output an "Error" level log message that dictates the nature of the failure. This Error log message would contain the specific processor that produced the ERROR as well as information on the specific FlowFile that was routed to failure and the nature of the failure.

It is possible to build a dataflow that monitors NiFi's nifi-app.log (TailFile processor) for ERROR log messages, parses out the relevant information and pass that along to some monitoring system.

Thanks,

Matt

[#9]

Thanks @Matt, parsing the nifi-app.log sounds like a better approach to capture errors at each processor and be able to know where the flowfile failed. Perhaps Splunk might be a good fit for this task.