How to get alerts when a processor fails

Question

My Nifi Flow starts with capturechangeMysql processor. I am trying to enhance my flows by adding email alerts whenever a processor fails. I understand I can use PutEmail for this purpose. my flow fails most of the time at the capturechangeMysql because of Binlogs or connections or some other random issues. My thinking is to create a link out of failure relationship of this processor to PutEmail, But there is no failure relationship for this processor. (It only has success relation.)

In this scenario how can i alert when there is a failure in capturechangeMysql processor ?

Please share your thoughts or if i am missing something or other alternatives.

cotopaul · Accepted Answer

@SandyClouds,You are correct when saying that you have no failure queue from CaptuerChangeMySQL and you are not missing anything 🙂I do not know the perfect way to solve your issue, but for me, I have used the following two scenarios to achieve what you are looking for:1) REST API to Bulletin Board and extract only the messages generated for the ID of your CaptuerChangeMySQL.To achieve this, you will need an InvokeHTTP Processor, which will cll your REST API and extract everything from your bulletin board. Next, you can use an EvaluateJsonPathto extract what you want from the logs (for example $.bulletinBoard.bulletins). Next, you can use a SplitJSON to split every bulletin message into single FlowFiles. Afterwards, you add a new EvaluateJsonPath where you extract the error level, error message, error source name and the timestamp and save them as attributes ($.bulletin.level , $.bulletin.message, $.bulletin.sourceName, $.bulletin.timestamp). Using an RouteOnAttribute, you then filter out your Error Messages generated by your CaptuerChangeMySQL Processor and send them by email.2) TailFile on your nifi-app.logs and filter out all the error messages generated by your CaptuerChangeMySQL Processor. The results are then sent per email.Again, not the best solution but at least this got the things working for me 🙂

steven-matison · Answer

@SandyClouds ^^ that is how you do it.

One important thing to mention. That processor is not meant to be deployed in a set it up and forget it situation. CDC against another database requires a lot more technical attention to how the data is changing over time AFTER you grab data for first time. In my opinion, this processor is meant to be a conversation starter, or a way to take a 1 time shot of data source, where you may be watching it run, but not expecting it to run indefinitely and keep 2 systems synced.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.