Support Questions
Find answers, ask questions, and share your expertise

Monitoring processing time for flowfiles in NiFi

New Contributor

Hi,


I am looking for a way to monitor flow files as they are ingested into the start of the flow and end up being published to Kafka. I have looked at the following link (https://pierrevillard.com/2017/05/15/monitoring-nifi-workflow-sla/), but It's not exactly what I am looking for.


I would basically like to monitor the flow files as they go trough each processor and if they take longer than a certain amount of time to reach a certain point, send out a mail notification.


I was thinking of adding a RouteOnAttribute in multiple locations in the flow, but I am not sure if this is the best way to do this. I am also not sure if the time the flowfile came in, will be kept as an attribute throughout the whole flow or if a new time is generate each time the flow file hits a new processor.

3 REPLIES 3

Super Guru

You can use the SiteToSiteProvenanceReportingTask for this. Filter the reporting task to only emit events at the "certain point" you mention above. Each event has a "timestampMillis" and "lineageStart" field, you should be able to route on the difference of the two using QueryRecord, with something like:

SELECT * FROM FLOWFILE WHERE timestampMillis - lineageStart > 60000

Which should emit a flowfile containing all events for which the associated entity (in this case, the flow file in the system) has been in the flow for over a minute.

Explorer

@mburgess is there a way i can access the lineage duration from a update attribute processor. I like to store this in a database along with other meta information for analysis. 

Moderator

@Althotta As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks!


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
; ;