Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to identify source processor for failed NiFi flowfiles written to a log

Solved Go to solution

How to identify source processor for failed NiFi flowfiles written to a log

Rising Star

Hello,

We're logging failed flowfiles to local file system; when reviewing the failed flowfiles, there is no information to tell us at which processor a flowfile failed; I was thinking about tagging the flowfiles at each processor, with the processor name it's about to enter; and when logging the failed flowfile, embed this tag/attribute into the file name; so looking at the file name you could tell at what processor the flowfile failed; I'm sure this is not the best way, so any suggestions on how to do this better.

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to identify source processor for failed NiFi flowfiles written to a log

Master Guru

@Raj B

Not all NiFi Processors will write attributes to FlowFiles about failures or errors. The documentation for each processor should include what attributes are written by that processor and what information those attributes will contain. There is no global enforcement by the NiFi controller on what attributes a processor must create. This is completely in the control of the developer who wrote each processor.

That being said, it is good practice that any processor that has a "Failure" relationship should output an "Error" level log message that dictates the nature of the failure. This Error log message would contain the specific processor that produced the ERROR as well as information on the specific FlowFile that was routed to failure and the nature of the failure.

It is possible to build a dataflow that monitors NiFi's nifi-app.log (TailFile processor) for ERROR log messages, parses out the relevant information and pass that along to some monitoring system.

Thanks,

Matt

8 REPLIES 8

Re: How to identify source processor for failed NiFi flowfiles written to a log

New Contributor
@Raj B

You can do this in many ways,

1. You can have an update attribute processor at the beginning of every Processor Group to specify the module name. But this cannot be replicated at processor level. Here, you keep updating the same attribute throughout the flow

2. You have have a 'Metadata' -> as a table or a flat file that will log all the attributes of the flow file. Please check AttributeToJson processor. This can log flow attributes both as an attribute or content. Also, note that every nifi processor adds additional 'error' attribute in case of any error which can be captured by AttributeToJson processor.

Hope this helps

Re: How to identify source processor for failed NiFi flowfiles written to a log

Rising Star

Thanks @Balakrishnan Ramasamy; didn't know about the 'error' attribute on failed flowfiles.

Re: How to identify source processor for failed NiFi flowfiles written to a log

Rising Star

Hi @Balakrishnan Ramasamy, do you know where I can find documentation about the 'error' attribute; I was looking in the NiFi documentation and it does not show the 'error' as an available attribute. Thanks.

11867-flowfile-core-attributes.png

Re: How to identify source processor for failed NiFi flowfiles written to a log

New Contributor
@Raj B

Please check for the 'Writes Attribute' in 'usage' of the processor. It describes the attribute each processor writes in case of failure. Below is the Usage of 'ExecuteStreamCommand' and it details 4 different error it writes during the execution.

11927-writes-attribute.jpg

Re: How to identify source processor for failed NiFi flowfiles written to a log

Rising Star

Thank you, I never paid attention to the "Writes Attributes" and didn't know what they were for.

Highlighted

Re: How to identify source processor for failed NiFi flowfiles written to a log

New Contributor

writes attribute of some processors (SplitRecord, ReplaceText ) does not contains the error it writes during execution. 

How and where to identify the error in that case? 

Re: How to identify source processor for failed NiFi flowfiles written to a log

Master Guru

@Raj B

Not all NiFi Processors will write attributes to FlowFiles about failures or errors. The documentation for each processor should include what attributes are written by that processor and what information those attributes will contain. There is no global enforcement by the NiFi controller on what attributes a processor must create. This is completely in the control of the developer who wrote each processor.

That being said, it is good practice that any processor that has a "Failure" relationship should output an "Error" level log message that dictates the nature of the failure. This Error log message would contain the specific processor that produced the ERROR as well as information on the specific FlowFile that was routed to failure and the nature of the failure.

It is possible to build a dataflow that monitors NiFi's nifi-app.log (TailFile processor) for ERROR log messages, parses out the relevant information and pass that along to some monitoring system.

Thanks,

Matt

Re: How to identify source processor for failed NiFi flowfiles written to a log

Rising Star

Thanks @Matt, parsing the nifi-app.log sounds like a better approach to capture errors at each processor and be able to know where the flowfile failed. Perhaps Splunk might be a good fit for this task.

Don't have an account?
Coming from Hortonworks? Activate your account here