Skip to main content
Raj_B
Raj_B
Gold ⭐️
January 24, 2017
Solved

How to Identify the Source Processor Responsible for Failed FlowFiles in Apache NiFi

Hello,

We're logging failed flowfiles to local file system; when reviewing the failed flowfiles, there is no information to tell us at which processor a flowfile failed; I was thinking about tagging the flowfiles at each processor, with the processor name it's about to enter; and when logging the failed flowfile, embed this tag/attribute into the file name; so looking at the file name you could tell at what processor the flowfile failed; I'm sure this is not the best way, so any suggestions on how to do this better.

Thanks.

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
January 31, 2017

@Raj B

Not all NiFi Processors will write attributes to FlowFiles about failures or errors. The documentation for each processor should include what attributes are written by that processor and what information those attributes will contain. There is no global enforcement by the NiFi controller on what attributes a processor must create. This is completely in the control of the developer who wrote each processor.

That being said, it is good practice that any processor that has a "Failure" relationship should output an "Error" level log message that dictates the nature of the failure. This Error log message would contain the specific processor that produced the ERROR as well as information on the specific FlowFile that was routed to failure and the nature of the failure.

It is possible to build a dataflow that monitors NiFi's nifi-app.log (TailFile processor) for ERROR log messages, parses out the relevant information and pass that along to some monitoring system.

Thanks,

Matt

View in thread

8 replies

rbalakrishnantc
rbalakrishnantc
Silver ⭐️⭐️
January 25, 2017
@Raj B

You can do this in many ways,

1. You can have an update attribute processor at the beginning of every Processor Group to specify the module name. But this cannot be replicated at processor level. Here, you keep updating the same attribute throughout the flow

2. You have have a 'Metadata' -> as a table or a flat file that will log all the attributes of the flow file. Please check AttributeToJson processor. This can log flow attributes both as an attribute or content. Also, note that every nifi processor adds additional 'error' attribute in case of any error which can be captured by AttributeToJson processor.

Hope this helps

Translate
Raj_B
Raj_BAuthor
Gold ⭐️
January 26, 2017

Thanks @Balakrishnan Ramasamy; didn't know about the 'error' attribute on failed flowfiles.

Translate
rbalakrishnantc
rbalakrishnantc
Silver ⭐️⭐️
January 31, 2017
@Raj B

Please check for the 'Writes Attribute' in 'usage' of the processor. It describes the attribute each processor writes in case of failure. Below is the Usage of 'ExecuteStreamCommand' and it details 4 different error it writes during the execution.

11927-writes-attribute.jpg

Translate
    Raj_B
    Raj_BAuthor
    Gold ⭐️
    February 1, 2017

    Thank you, I never paid attention to the "Writes Attributes" and didn't know what they were for.

    Translate
    MattWho
    Titanium ⭐️⭐️⭐️
    MattWhoAnswer
    Titanium ⭐️⭐️⭐️
    January 31, 2017

    @Raj B

    Not all NiFi Processors will write attributes to FlowFiles about failures or errors. The documentation for each processor should include what attributes are written by that processor and what information those attributes will contain. There is no global enforcement by the NiFi controller on what attributes a processor must create. This is completely in the control of the developer who wrote each processor.

    That being said, it is good practice that any processor that has a "Failure" relationship should output an "Error" level log message that dictates the nature of the failure. This Error log message would contain the specific processor that produced the ERROR as well as information on the specific FlowFile that was routed to failure and the nature of the failure.

    It is possible to build a dataflow that monitors NiFi's nifi-app.log (TailFile processor) for ERROR log messages, parses out the relevant information and pass that along to some monitoring system.

    Thanks,

    Matt

    Translate
      Raj_B
      Raj_BAuthor
      Gold ⭐️
      February 1, 2017

      Thanks @Matt, parsing the nifi-app.log sounds like a better approach to capture errors at each processor and be able to know where the flowfile failed. Perhaps Splunk might be a good fit for this task.

      Translate
      Terms & ConditionsAccessibility statement