It appears that Nifi has not standardised how exceptions/errors are propagated to failure queues? For instance PublishAMQP simply doesn't send anything out to the failure queue if the broker is not available; ExecuteSQL attaches an executesql.error.message if it fails; and InvokeHTTP attaches a invokehttp.java.exception.message; ConvertJsonToSQL simply writes to the failure queue but doesn't actual give any indication of the failure. Now, We have an on-boarding process; we split a csv -> we create a json payload -> we put it on a queue, and write it to a database (plus a few other things). Now, if something goes I need to do a couple of things. If PublishAMQP fails I need to create a 'Case' in Flowable so it can be looked into; at present the only way I see in achieving this is to tail the nifi-app log looking for some kind of error related to the queue? Now when using ConvertJSONToSQL you get no indication of what the error is (no attribute set), when it's written to the failure queue - instead you need to look at the nifi-app log again. Now I need to start a manual intervention Flowable BPMN taking the record in the failure queue without no idea what the error is as it's in the nifi-app.log making this a useless process. To get an understanding within the manual intervention process at some point I need to (enrich) go lookup the failure which I'd need to store somewhere once I've got it out of the app-log (storing it somewhere else) as we purge weekly, and manual workflows cha go on for sometime especially a KYC process. As I don't have the exception ID I'd need to do a text search in my new error log table as exceptions can differ based on my uuid? Then attach it the workflow. Firstly, it appears that all failures in the app-log have an Id why not simply attached that as a standard identifier for correlation if moving to the failure queue; or simply writing what's written in the app-log error line; Seems simply but then again maybe not? Is there a mechanism in-place to capture exceptions as they are raised. We've though about writing a custom error log listener and sending the errors to another place - which just sounds bizarre given that all the information is with Nifi lol. Secondly, am I not using Nifi for it's intended purposes? We do not want to have an operator staring at the screen in case 'something' goes wrong. Ron.
... View more
