Monitoring processing time for flowfiles in NiFi

[#1]

Hi,

I am looking for a way to monitor flow files as they are ingested into the start of the flow and end up being published to Kafka. I have looked at the following link (https://pierrevillard.com/2017/05/15/monitoring-nifi-workflow-sla/), but It's not exactly what I am looking for.

I would basically like to monitor the flow files as they go trough each processor and if they take longer than a certain amount of time to reach a certain point, send out a mail notification.

I was thinking of adding a RouteOnAttribute in multiple locations in the flow, but I am not sure if this is the best way to do this. I am also not sure if the time the flowfile came in, will be kept as an attribute throughout the whole flow or if a new time is generate each time the flow file hits a new processor.

[#2]

You can use the SiteToSiteProvenanceReportingTask for this. Filter the reporting task to only emit events at the "certain point" you mention above. Each event has a "timestampMillis" and "lineageStart" field, you should be able to route on the difference of the two using QueryRecord, with something like:

SELECT * FROM FLOWFILE WHERE timestampMillis - lineageStart > 60000

Which should emit a flowfile containing all events for which the associated entity (in this case, the flow file in the system) has been in the flow for over a minute.

[#3]

@mburgess is there a way i can access the lineage duration from a update attribute processor. I like to store this in a database along with other meta information for analysis. 

[#4]

@Althotta As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks!