How to handle error in NIFI flow and send notification email on flow successfully completed or failure?

[#1]

Hello,

I have design the nifi flow like this

selectHiveQL->convertAvroToJson->SplitJson->EvaluateJsonPath->RoutOnAttribute(depends on condition the query goes on)

->presto or ->ReplaceText->putHiveQL

hive table is

query_id job_type query

1 presto create table hive.db.table as select * from mysql.db.table

2 hive ALTER table db.table set tblproperties ("orc.create.index"="true")

3 hive ALTER table db.table set tblproperties ("transactional"="true")

4 presto create table hive.db.table_stage as select * from mysql.db.table

selectHiveQL-select query on hive table contain query_id, job_type and query

convertAvroToJson->convert avro format into json

SplitJson-split the record

EvaluateJsonPath-set the query_id, job_type and query attribute

RoutOnAttribute-depending on condition of job_type that query is executed in presto or hive

presto-custom processor executed the query

ReplaceText-set the putHiveQL query

putHiveQL-execute hive query

complete flow-flow.png

problems:

1) How to stop the flow if any query failed from any processor.

2) I have tried to connect the presto failure to presto processor that time if query failed the next query not goes to presto processor but failed query is continuously executed in presto processor.

3) if the query is alternate and presto query failed that time hive query goes to hive processor(backpressure of hive failure also set to 1).

4) How to send email notification if the flow is successfully completed or failed?

Please provide your suggestions .

Thanks in advance .

[#2]

Hi Mitthu,

Here is an article I wrote about handling failures in NiFi: https://community.hortonworks.com/articles/76598/nifi-error-handling-design-pattern-1.html

It describes how to retry failures X times, then send an email, then wait for administrative input. This might help you address the requirements of your solution.

You could also add a PutEmail processor on the "Success" relationship to send an email after processing succeeds.