Multiple notifications are triggering for both failure/success in Nifi flow

[#1]

I'm trying to copy data from source table to destination table and as part of the process I'm setting the partition size as 100k records. There are 10 Millions records in the source table. So the number of flow files generating are 100. I'm adding a put email at the end of the flow and it is sending 100 mails(1 mail per FlowFile). However I want send mail notification only once that to after the process of data copy completed successfully. This is the same case I observed during failure as well. Email is sending once per Flow file. I want to restrict it to once after the processor completed(successfully/fail). Could someone help me on this please.

[#2 · Accepted solution]

@Pavan M

You can use MergeContent processor before PutEmail processor and configure maximum group size to 100 then processor will wait until the maximum group size then it will keep give only one flowfile after merging the 100 flowfiles.

Then use Merged relationship feed to PutEmail processor in this case we are going to wait until 100Flowfiles are reached to destination then merge content merges all the flowfiles into one flowfile and put email processor will receive only one flowfile,You are going to have one email notification instead of 100.

if you are not sure about number of flowfiles then make use of Max Bin Age Property, this property will force to merge all the flowfiles, you need to keep maximum number of entries property,Maximum Group Size according to your flowfiles and total size of all flowfiles.

Flow:-

1.PutSQL Processor
2.MergeContent
3.PutEmail

Please refer to below link for configuring merge content processor

https://community.hortonworks.com/questions/149047/nifi-how-to-handle-with-mergecontent-processor.ht...
https://community.hortonworks.com/questions/64337/apache-nifi-merge-content.html

(or)

If you want only one email when first flowfile reaches into destination instead of waiting for mergecontent processor to merge all 100 flowfiles.

Then use control rate processor change the configurations

Rate Control Criteria

flowfile count

Maximum Rate

1

Rate Controlled Attribute No value set Time Duration

10 min //change the number according to your putsql processor time taken to write records to destination.

so now we are releasing one flowfile from Control Rate processor for every 10 minutes.

Then the queue configuration before control rate processor change the Flowfile expiration to some small number like 10 Sec.

With this configurations we are going to release first flowfile from control rate processor and then first flowfile reaches to PutEmail processor, we will get one notification.

All the rest 99 flowfiles are waiting in the queue before Control Rate Processor and they are going to expire in 10 sec after they reached in to the queue.

By following this approach you can get notification from first flowfile once it reaches to destination

Flow:-

1.PutSQL processor
2.Control Rate
3.PutEmail

[#3]

Thanks for the response @Shu,

I will try the options suggested by you and will get back.