Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Solved Go to solution

Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Rising Star

Hello, I'm looking for some ideas on getting notified when a NiFi job is completed, i.e. processed all the data from the source.

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Hi @Raj B,

There is not really the notion of "completed" in NiFi. We are in a streaming approach, and there is no a priori knowledge of a start and a end. However you can find some workarounds based on your use case, for example: you could add a MonitorActivity processor to notify you if there is no more event processed after X minutes. Anyway, there is no general approach, it is really tied to your use case.

Hope this helps.

6 REPLIES 6

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Hi @Raj B,

There is not really the notion of "completed" in NiFi. We are in a streaming approach, and there is no a priori knowledge of a start and a end. However you can find some workarounds based on your use case, for example: you could add a MonitorActivity processor to notify you if there is no more event processed after X minutes. Anyway, there is no general approach, it is really tied to your use case.

Hope this helps.

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Rising Star

@Pierre Villard I understand there's no completion when we're streaming. But if we're moving static set of data from a database to HDFS or from Unix to HDFS/Hive, it would be handy if you can get an alert saying the job is complete. MonitorActivity processor would work for my use case. Thanks.

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Expert Contributor

How will you find out all flowfiles from a source have been processed? If you can get that figured out nifi has several components ex. put email, which can send an email to given recipients. You can also set up an SNMP agent, use setSNMP to set a message , which the snmp agent can forward to recipients.

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Rising Star

as @Pierre Villard mentioned above, MonitorActivity processor can tell you when there is no activity for a specified period of time.

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Maybe Wait/Notify processor might be useful for some use cases, too. It will support waiting for multiple signals from next version of NiFi. There's a sample flow image, a part of flow waits for other part of flow send notification signal:

https://github.com/apache/nifi/pull/1420

Re: Ideas on how to get notified when NiFi completed processing all flowfiles from the Source

Rising Star

thank you @kkawamura for the heads up on the new processor.

Don't have an account?
Coming from Hortonworks? Activate your account here