Member since
01-21-2022
6
Posts
1
Kudos Received
0
Solutions
02-23-2022
12:41 AM
Hi, @RonMilne , A claim can contain one or more flowfiles and NiFi uses it to optimize storage for the content repository. It's totally possible that input and output flowfiles of a processor belong in the same claim. Please see this article for an explanation of what claims are and how they are managed. Cheers, André
... View more
02-06-2022
04:18 PM
You can try the following Jolt spec in a JoltTransformRecord processor to change the name of one or more columns: Jolt Transformation DSL: Chain Jolt Specification: [ { "operation": "shift", "spec": { "*": "&", "surname": "lastname" } }, { "operation": "remove", "spec": { "surname": "" } } ]
... View more
01-31-2022
11:52 PM
Does anyone know where to find the documentation that details exactly how to achieve this? Regards, Ron.
... View more
Labels:
- Labels:
-
Apache NiFi
01-28-2022
04:12 PM
It appears that Nifi has not standardised how exceptions/errors are propagated to failure queues? For instance PublishAMQP simply doesn't send anything out to the failure queue if the broker is not available; ExecuteSQL attaches an executesql.error.message if it fails; and InvokeHTTP attaches a invokehttp.java.exception.message; ConvertJsonToSQL simply writes to the failure queue but doesn't actual give any indication of the failure. Now, We have an on-boarding process; we split a csv -> we create a json payload -> we put it on a queue, and write it to a database (plus a few other things). Now, if something goes I need to do a couple of things. If PublishAMQP fails I need to create a 'Case' in Flowable so it can be looked into; at present the only way I see in achieving this is to tail the nifi-app log looking for some kind of error related to the queue? Now when using ConvertJSONToSQL you get no indication of what the error is (no attribute set), when it's written to the failure queue - instead you need to look at the nifi-app log again. Now I need to start a manual intervention Flowable BPMN taking the record in the failure queue without no idea what the error is as it's in the nifi-app.log making this a useless process. To get an understanding within the manual intervention process at some point I need to (enrich) go lookup the failure which I'd need to store somewhere once I've got it out of the app-log (storing it somewhere else) as we purge weekly, and manual workflows cha go on for sometime especially a KYC process. As I don't have the exception ID I'd need to do a text search in my new error log table as exceptions can differ based on my uuid? Then attach it the workflow. Firstly, it appears that all failures in the app-log have an Id why not simply attached that as a standard identifier for correlation if moving to the failure queue; or simply writing what's written in the app-log error line; Seems simply but then again maybe not? Is there a mechanism in-place to capture exceptions as they are raised. We've though about writing a custom error log listener and sending the errors to another place - which just sounds bizarre given that all the information is with Nifi lol. Secondly, am I not using Nifi for it's intended purposes? We do not want to have an operator staring at the screen in case 'something' goes wrong. Ron.
... View more
Labels:
- Labels:
-
Apache NiFi
01-27-2022
12:43 AM
I want to add a TailFile processor to tail nifi-app.log - I'll eventually be storing all [ERROR] to a database, but first I have no idea what values to uses as I've noticed the file name rotates? Can someone shed some light on what attribute values I need to set? Greatly appreciated. Ron.
... View more
Labels:
- Labels:
-
Apache NiFi
01-25-2022
01:28 PM
@RonMilne I recommend taking your initial CSV record file and partitioning by the " salesRepId" in to multiple new JSON records. This can be accomplished using the PartitionRecord processor utilizing a CSVReader and a JsonRecordSetWriter. Your PartitionRecord processor configuration would look like this: Your CSVReader would be configured something like this (you'll need to modify it for your specific record's Schema: Note: Pop-out shows the "Schema Text" property and don't forget to set "Treat First Line as Header" property to "true" The JsonRecordSetWriter would need to be configured to produce the desired JSON record output format. However, just leaving default configuration will out put a separate FlowFile for each unique "SalesRepId". If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more