Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

[#1]

Hello, I'm looking for some ideas on getting notified when a NiFi job is completed, i.e. processed all the data from the source.

Thanks.

[#2 · Accepted solution]

Hi @Raj B,

There is not really the notion of "completed" in NiFi. We are in a streaming approach, and there is no a priori knowledge of a start and a end. However you can find some workarounds based on your use case, for example: you could add a MonitorActivity processor to notify you if there is no more event processed after X minutes. Anyway, there is no general approach, it is really tied to your use case.

Hope this helps.

[#3]

@Pierre Villard I understand there's no completion when we're streaming. But if we're moving static set of data from a database to HDFS or from Unix to HDFS/Hive, it would be handy if you can get an alert saying the job is complete. MonitorActivity processor would work for my use case. Thanks.

[#4]

How will you find out all flowfiles from a source have been processed? If you can get that figured out nifi has several components ex. put email, which can send an email to given recipients. You can also set up an SNMP agent, use setSNMP to set a message , which the snmp agent can forward to recipients.

[#5]

as @Pierre Villard mentioned above, MonitorActivity processor can tell you when there is no activity for a specified period of time.

[#6]

Maybe Wait/Notify processor might be useful for some use cases, too. It will support waiting for multiple signals from next version of NiFi. There's a sample flow image, a part of flow waits for other part of flow send notification signal:

https://github.com/apache/nifi/pull/1420

[#7]

thank you @kkawamura for the heads up on the new processor.