question Re: [Nifi] - Multiple predicate in recordpath filter in Support Questions

[Nifi] - Multiple predicate in recordpath filter

stephane_davy — Mon, 30 Nov 2020 16:37:50 GMT

Hello all,

I'd like to use the updateRecord processor to create some fields in my JSON. But I'd like to use some conditions to do that. In the recordpath documentation, I can see that there is a "filter" capability, which works like this: /field[filter]/path, but as far as I can see it's not possible to have multiple conditions in this filter. I'd like to do something like:

if (field1 == A or field1 == B) { field2 = C } else if (field1 == D and field1 != E) { field2 = G }

The example is silly but this is to give you the idea. By the way, except using a script, I don't know how to do that, except may be extracting the fields, put it as attribute and play with expression language but this would mean working with single-record flows and I don't want to do that.

Any idea?

Thanks for your suggestions

Stéphane

Re: [Nifi] - Multiple predicate in recordpath filter

justenji — Thu, 03 Dec 2020 11:04:28 GMT

@stephane_davy Maybe you want to give this solution a try?
Hope the explanation is clear. If not feel free to ask again.
<Picture removed>

Re: [Nifi] - Multiple predicate in recordpath filter

justenji — Thu, 03 Dec 2020 11:01:37 GMT

@stephane_davy Sorry, my above picture is a kind of vision test...

Flow and Step1 and 2

Step 3 Rule 1Step 3 Rule 2Step 4

Re: [Nifi] - Multiple predicate in recordpath filter

stephane_davy — Mon, 07 Dec 2020 16:37:38 GMT

Hello @justenji

Thanks for you reply. The problem here is that you are working with attributes and expression language, which would mean to have one flowfile per json object. I'm trying to have a record-oriented flow-file, which means that everything happens with recordpath and / or Jolt transformations. My flow file have multiple JSON records and as such attributes would take multiple values. And I don't want to split my records, this is really resource consuming.

Best regards,

Stéphane

Re: [Nifi] - Multiple predicate in recordpath filter

justenji — Tue, 08 Dec 2020 07:54:53 GMT

Hello@stephane_davy

I'm sorry, I haven't seen that record-processing is a "must".

Because I haven't much experience with this kind of syntax I sadly will not be able to help you here.

But if you find a solution just be so kind and show it all of us here. Thank you!

Re: [Nifi] - Multiple predicate in recordpath filter

stephane_davy — Tue, 08 Dec 2020 16:24:57 GMT

Hello @justenji

Thanks anyway to have taken some time to look after my issue. For now, the only solution I have found is a groovy script.

Stéphane

Re: [Nifi] - Multiple predicate in recordpath filter

stephane_davy — Wed, 09 Dec 2020 10:12:15 GMT

I need to test this, but actually it seems that the queryRecord is exactly why I need. It is possible to build some complex conditions using SQL-like language on the record content and then perform some routing based on that

Re: [Nifi] - Multiple predicate in recordpath filter

justenji — Wed, 09 Dec 2020 12:04:48 GMT

@stephane_davy
I'm working in the same direction because I couldn't believe that there is no possibility.

But I'm still struggling with problems of the correct MergeRecord.
So we both go the same way.

Re: [Nifi] - Multiple predicate in recordpath filter

justenji — Mon, 14 Dec 2020 08:18:38 GMT

@stephane_davy
Sorry I couldn't work on this further last week. Now my solution looks like this.

JSON from GenerateFlowFile

[ { "myfield": "JustForHavingJson", "myfield1": "A" }, { "myfield": "JustForHavingJson", "myfield1": "B" }, { "myfield": "JustForHavingJson", "myfield1": "C" }, { "myfield": "JustForHavingJson", "myfield1": "D" }, { "myfield": "JustForHavingJson", "myfield1": "E" }, { "myfield": "JustForHavingJson", "myfield1": "X" }, { "myfield": "JustForHavingJson", "myfield1": "" } ]

Definitions of ControllerServices and AvroSchemaRegistry
ControllerServices and AvroSchemaRegistry
Flow
Flow

Details of RecordProcessing-Processors
Details RecordProcessing-Processors

Result FF-Content after MergeRecord:

{"myfield":"JustForHavingJson","myfield1":"A","myfield2":"C"} {"myfield":"JustForHavingJson","myfield1":"B","myfield2":"C"} {"myfield":"JustForHavingJson","myfield1":"C","myfield2":""} {"myfield":"JustForHavingJson","myfield1":"E","myfield2":""} {"myfield":"JustForHavingJson","myfield1":"X","myfield2":""} {"myfield":"JustForHavingJson","myfield1":"","myfield2":""} {"myfield":"JustForHavingJson","myfield1":"D","myfield2":"G"}

Personally I wouldn't do a MergeRecord at the end but go on with the three single connections of the UpdateRecord processors.

Do you think this could be a possible solution or have you found a better way to do this?

Re: [Nifi] - Multiple predicate in recordpath filter

stephane_davy — Mon, 14 Dec 2020 15:23:24 GMT

Hello @justenji

Thanks for the detailed explanation, this is also what I have tested on my side and in my opinion this does the job being "record oriented" 🙂

Have a nice day