How to get alerts when a processor fails

[#1]

My Nifi Flow starts with capturechangeMysql processor. I am trying to enhance my flows by adding email alerts whenever a processor fails. I understand I can use PutEmail for this purpose. my flow fails most of the time at the capturechangeMysql because of Binlogs or connections or some other random issues. My thinking is to create a link out of failure relationship of this processor to PutEmail, But there is no failure relationship for this processor. (It only has success relation.)

In this scenario how can i alert when there is a failure in capturechangeMysql processor ?

Please share your thoughts or if i am missing something or other alternatives.

[#2 · Accepted solution]

@SandyClouds,

You are correct when saying that you have no failure queue from CaptuerChangeMySQL and you are not missing anything 🙂

I do not know the perfect way to solve your issue, but for me, I have used the following two scenarios to achieve what you are looking for:

1) REST API to Bulletin Board and extract only the messages generated for the ID of your CaptuerChangeMySQL.

To achieve this, you will need an InvokeHTTP Processor, which will cll your REST API and extract everything from your bulletin board. Next, you can use an EvaluateJsonPathto extract what you want from the logs (for example $.bulletinBoard.bulletins). Next, you can use a SplitJSON to split every bulletin message into single FlowFiles. Afterwards, you add a new EvaluateJsonPath where you extract the error level, error message, error source name and the timestamp and save them as attributes ($.bulletin.level , $.bulletin.message, $.bulletin.sourceName, $.bulletin.timestamp). Using an RouteOnAttribute, you then filter out your Error Messages generated by your CaptuerChangeMySQL Processor and send them by email.

2) TailFile on your nifi-app.logs and filter out all the error messages generated by your CaptuerChangeMySQL Processor. The results are then sent per email.

Again, not the best solution but at least this got the things working for me 🙂

[#3]

Hey @cotopaul Thanks for your reply.

ohh disappointing from Nifi.. 😞 

But I am not sure why this limitation for this processor.

Out of your 2 options i am interested in 1st one. May be I can also query the API for Processors status (

"/processors/run-status-details/queries

Submits a query to retrieve the run status details of all processors that are in the given list of Processor IDs") instead of bulletin board.

Can you please share the Nifi flow template for option 1 ?

Thanks for your hacky ways. 🙂 

[#4]

@SandyClouds,

I do not have a template because I no longer have access to that project but I have provided you will all the info you need to develop your own system 🙂

And it mostly depends on your use case. You can use any API you require for your use case. But in terms of processors your require the following: InvokeHTTP to perform the API Call, EvaluateJSONPath + SplitJSON to extract the relevant lines out of your bulletin boards and RouteOnAttribute to identify the errors you need. Other than that, your imagination is your best friend.

[#5]

@SandyClouds  ^^ that is how you do it.

One important thing to mention.  That processor is not meant to be deployed in a set it up and forget it situation.   CDC against another database requires a lot more technical attention to how the data is changing over time AFTER you grab data for first time.   In my opinion, this processor is meant to be a conversation starter, or a way to take a 1 time shot of data source, where you may be watching it run, but not expecting it to run indefinitely and keep 2 systems synced.

[#6]

Thanks for pointing out. @steven-matison 

Are you trying to mention that if the source DB change some column formats, add/delete new/old columns  ? (I am taking care of these, in the sense, whenever source changes structure, destination is already mimicking it outside Nifi.) 

So ideally i was thinking this will work indefinitely and they will be in sync. and both of the DBs are mysql. 

If my thinking is wrong, can you please enlighten me on why i should not relay on this kind of syncing ?  What kind of issues are you anticipating in long run ?