Member since
01-21-2022
6
Posts
1
Kudos Received
0
Solutions
02-02-2022
10:54 PM
If you use partition record (such as going from csv to json) and grouping by a column say /something the output contains the split in json - however, the provenance input and output claims are the same (it's the original data from the input in my case the csv contents)? Regards, Ron.
... View more
Labels:
- Labels:
-
Apache NiFi
02-02-2022
10:24 PM
1 Kudo
For the JoltTransformation what people will need to see is the data being fed to it in the context of the flow, i.e., the queue prior to the transform contains what? If it's an array then you're transform is wrong but seeing that will hopefully allow people to solve your issue. Regards, Ron.
... View more
01-31-2022
11:52 PM
Does anyone know where to find the documentation that details exactly how to achieve this? Regards, Ron.
... View more
Labels:
- Labels:
-
Apache NiFi
01-28-2022
04:12 PM
It appears that Nifi has not standardised how exceptions/errors are propagated to failure queues? For instance PublishAMQP simply doesn't send anything out to the failure queue if the broker is not available; ExecuteSQL attaches an executesql.error.message if it fails; and InvokeHTTP attaches a invokehttp.java.exception.message; ConvertJsonToSQL simply writes to the failure queue but doesn't actual give any indication of the failure. Now, We have an on-boarding process; we split a csv -> we create a json payload -> we put it on a queue, and write it to a database (plus a few other things). Now, if something goes I need to do a couple of things. If PublishAMQP fails I need to create a 'Case' in Flowable so it can be looked into; at present the only way I see in achieving this is to tail the nifi-app log looking for some kind of error related to the queue? Now when using ConvertJSONToSQL you get no indication of what the error is (no attribute set), when it's written to the failure queue - instead you need to look at the nifi-app log again. Now I need to start a manual intervention Flowable BPMN taking the record in the failure queue without no idea what the error is as it's in the nifi-app.log making this a useless process. To get an understanding within the manual intervention process at some point I need to (enrich) go lookup the failure which I'd need to store somewhere once I've got it out of the app-log (storing it somewhere else) as we purge weekly, and manual workflows cha go on for sometime especially a KYC process. As I don't have the exception ID I'd need to do a text search in my new error log table as exceptions can differ based on my uuid? Then attach it the workflow. Firstly, it appears that all failures in the app-log have an Id why not simply attached that as a standard identifier for correlation if moving to the failure queue; or simply writing what's written in the app-log error line; Seems simply but then again maybe not? Is there a mechanism in-place to capture exceptions as they are raised. We've though about writing a custom error log listener and sending the errors to another place - which just sounds bizarre given that all the information is with Nifi lol. Secondly, am I not using Nifi for it's intended purposes? We do not want to have an operator staring at the screen in case 'something' goes wrong. Ron.
... View more
Labels:
- Labels:
-
Apache NiFi
01-27-2022
12:43 AM
I want to add a TailFile processor to tail nifi-app.log - I'll eventually be storing all [ERROR] to a database, but first I have no idea what values to uses as I've noticed the file name rotates? Can someone shed some light on what attribute values I need to set? Greatly appreciated. Ron.
... View more
Labels:
- Labels:
-
Apache NiFi
01-21-2022
08:44 PM
Our sales team are extracting the months sales opportunities, actuals etc. Within the one CSV file there are potentially 100+ rows. Now, we need to create new flow files (array?) containing individual sales people. We have 20+ sales people each working on many opportunities. salesOpId,salesRepId, salesOpStage, ..... 142435,135235,inception .... 142436,572856,inception .... 142437,135235,pipelining .... 142435,572856,inception.... 149468,135235,contract.... 149464,135653,contract.... As you can see salesRepId 135235 is working against 3 opportunities, (142436,142437,149468) salesRepId 572856 is working against 2 opportunities (142436, 142435) salesRepId 135653 is working on a single opportunity (149464) For this example I need 3 FlowFiles created (which will be sent to their respective managers); I'm capable of reading the CSV changing it to Json; but now I'm completely stumped on the next step - split the json? Any help would be greatly appreciated. Ron M.
... View more
Labels:
- Labels:
-
Apache NiFi