Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Monitoring processing time for flowfiles in NiFi

Monitoring processing time for flowfiles in NiFi

New Contributor

Hi,


I am looking for a way to monitor flow files as they are ingested into the start of the flow and end up being published to Kafka. I have looked at the following link (https://pierrevillard.com/2017/05/15/monitoring-nifi-workflow-sla/), but It's not exactly what I am looking for.


I would basically like to monitor the flow files as they go trough each processor and if they take longer than a certain amount of time to reach a certain point, send out a mail notification.


I was thinking of adding a RouteOnAttribute in multiple locations in the flow, but I am not sure if this is the best way to do this. I am also not sure if the time the flowfile came in, will be kept as an attribute throughout the whole flow or if a new time is generate each time the flow file hits a new processor.

1 REPLY 1

Re: Monitoring processing time for flowfiles in NiFi

You can use the SiteToSiteProvenanceReportingTask for this. Filter the reporting task to only emit events at the "certain point" you mention above. Each event has a "timestampMillis" and "lineageStart" field, you should be able to route on the difference of the two using QueryRecord, with something like:

SELECT * FROM FLOWFILE WHERE timestampMillis - lineageStart > 60000

Which should emit a flowfile containing all events for which the associated entity (in this case, the flow file in the system) has been in the flow for over a minute.