If you use partition record (such as going from csv to json) and grouping by a column say /something the output contains the split in json - however, the provenance input and output claims are the same (it's the original data from the input in my case the csv contents)?   Regards, Ron. ... View more

Does anyone know where to find the documentation that details exactly how to achieve this?   Regards, Ron. ... View more

It appears that Nifi has not standardised how exceptions/errors are propagated to failure queues? For instance PublishAMQP simply doesn't send anything out to the failure queue if the broker is not available; ExecuteSQL attaches an  executesql.error.message if it fails; and InvokeHTTP attaches a invokehttp.java.exception.message; ConvertJsonToSQL simply writes to the failure queue but doesn't actual give any indication of the failure.   Now, We have an on-boarding process; we split a csv -> we create a json payload -> we put it on a queue, and write it to a database (plus a few other things). Now, if something goes I need to do a couple of things. If PublishAMQP fails I need to create a 'Case' in Flowable so it can be looked into; at present the only way I see in achieving this is to tail the nifi-app log looking for some kind of error related to the queue?   Now when using ConvertJSONToSQL you get no indication of what the error is (no attribute set), when it's written to the failure queue - instead you need to look at the nifi-app log again. Now I need to start a manual intervention Flowable BPMN taking the record in the failure queue without no idea what the error is as it's in the nifi-app.log making this a useless process.   To get an understanding within the manual intervention process at some point I need to (enrich) go lookup the failure which I'd need to store somewhere once I've got it out of the app-log (storing it somewhere else) as we purge weekly, and manual workflows cha go on for sometime especially a KYC process. As I don't have the exception ID I'd need to do a text search in my new error log table as exceptions can differ based on my uuid? Then attach it the workflow.   Firstly, it appears that all failures in the app-log have an Id why not simply attached that as a standard identifier for correlation if moving to the failure queue; or simply writing what's written in the app-log error line; Seems simply but then again maybe not?   Is there a mechanism in-place to capture exceptions as they are raised. We've though about writing a custom error log listener and sending the errors to another place - which just sounds bizarre given that all the information is with Nifi lol.   Secondly, am I not using Nifi for it's intended purposes? We do not want to have an operator staring at the screen in case 'something' goes wrong.   Ron.   ... View more

I want to add a TailFile processor to tail nifi-app.log - I'll eventually be storing all [ERROR] to a database, but first I have no idea what values to uses as I've noticed the file name rotates? Can someone shed some light on what attribute values I need to set?   Greatly appreciated. Ron. ... View more