Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to identify source processor for failed NiFi flowfiles written to a log

avatar
Expert Contributor

Hello,

We're logging failed flowfiles to local file system; when reviewing the failed flowfiles, there is no information to tell us at which processor a flowfile failed; I was thinking about tagging the flowfiles at each processor, with the processor name it's about to enter; and when logging the failed flowfile, embed this tag/attribute into the file name; so looking at the file name you could tell at what processor the flowfile failed; I'm sure this is not the best way, so any suggestions on how to do this better.

Thanks.

1 ACCEPTED SOLUTION

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
8 REPLIES 8

avatar
Rising Star
@Raj B

You can do this in many ways,

1. You can have an update attribute processor at the beginning of every Processor Group to specify the module name. But this cannot be replicated at processor level. Here, you keep updating the same attribute throughout the flow

2. You have have a 'Metadata' -> as a table or a flat file that will log all the attributes of the flow file. Please check AttributeToJson processor. This can log flow attributes both as an attribute or content. Also, note that every nifi processor adds additional 'error' attribute in case of any error which can be captured by AttributeToJson processor.

Hope this helps

avatar
Expert Contributor

Thanks @Balakrishnan Ramasamy; didn't know about the 'error' attribute on failed flowfiles.

avatar
Expert Contributor

Hi @Balakrishnan Ramasamy, do you know where I can find documentation about the 'error' attribute; I was looking in the NiFi documentation and it does not show the 'error' as an available attribute. Thanks.

11867-flowfile-core-attributes.png

avatar
Rising Star
@Raj B

Please check for the 'Writes Attribute' in 'usage' of the processor. It describes the attribute each processor writes in case of failure. Below is the Usage of 'ExecuteStreamCommand' and it details 4 different error it writes during the execution.

11927-writes-attribute.jpg

avatar
Expert Contributor

Thank you, I never paid attention to the "Writes Attributes" and didn't know what they were for.

avatar
New Contributor

writes attribute of some processors (SplitRecord, ReplaceText ) does not contains the error it writes during execution. 

How and where to identify the error in that case? 

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

Thanks @Matt, parsing the nifi-app.log sounds like a better approach to capture errors at each processor and be able to know where the flowfile failed. Perhaps Splunk might be a good fit for this task.