question Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source in Archives of Support Questions (Read Only)

Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Raj_B — Thu, 19 Jan 2017 22:56:04 GMT

Hello, I'm looking for some ideas on getting notified when a NiFi job is completed, i.e. processed all the data from the source.

Thanks.

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

pvillard — Thu, 19 Jan 2017 23:01:46 GMT

Hi @Raj B,

There is not really the notion of "completed" in NiFi. We are in a streaming approach, and there is no a priori knowledge of a start and a end. However you can find some workarounds based on your use case, for example: you could add a MonitorActivity processor to notify you if there is no more event processed after X minutes. Anyway, there is no general approach, it is really tied to your use case.

Hope this helps.

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

knarayanan — Thu, 19 Jan 2017 23:03:09 GMT

How will you find out all flowfiles from a source have been processed? If you can get that figured out nifi has several components ex. put email, which can send an email to given recipients. You can also set up an SNMP agent, use setSNMP to set a message , which the snmp agent can forward to recipients.

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Raj_B — Fri, 20 Jan 2017 00:04:22 GMT

@Pierre Villard I understand there's no completion when we're streaming. But if we're moving static set of data from a database to HDFS or from Unix to HDFS/Hive, it would be handy if you can get an alert saying the job is complete. MonitorActivity processor would work for my use case. Thanks.

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Raj_B — Fri, 20 Jan 2017 00:06:03 GMT

as @Pierre Villard mentioned above, MonitorActivity processor can tell you when there is no activity for a specified period of time.

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

kkawamura — Fri, 20 Jan 2017 09:07:46 GMT

Maybe Wait/Notify processor might be useful for some use cases, too. It will support waiting for multiple signals from next version of NiFi. There's a sample flow image, a part of flow waits for other part of flow send notification signal:

https://github.com/apache/nifi/pull/1420

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Raj_B — Sun, 22 Jan 2017 09:22:58 GMT

thank you @kkawamura for the heads up on the new processor.