[Nifi] - Multiple predicate in recordpath filter

[#1]

Hello all,

I'd like to use the updateRecord processor to create some fields in my JSON. But I'd like to use some conditions to do that. In the recordpath documentation, I can see that there is a "filter" capability, which works like this: /field[filter]/path, but as far as I can see it's not possible to have multiple conditions in this filter. I'd like to do something like:

if (field1 == A or field1 == B) {

field2 = C

} else if (field1 == D and field1 != E) {
field2 = G
}

The example is silly but this is to give you the idea. By the way, except using a script, I don't know how to do that, except may be extracting the fields, put it as attribute and play with expression language but this would mean working with single-record flows and I don't want to do that.

Any idea?

Thanks for your suggestions

Stéphane

[#2]

@stephane_davy Maybe you want to give this solution a try?
Hope the explanation is clear. If not feel free to ask again.
<Picture removed>

[#3]

@stephane_davy   Sorry, my above picture is a kind of vision test...

Flow and Step1 and 2

Step 3 Rule 1Step 3 Rule 2Step 4

[#4]

Hello @justenji 

Thanks for you reply. The problem here is that you are working with attributes and expression language, which would mean to have one flowfile per json object. I'm trying to have a record-oriented flow-file, which means that everything happens with recordpath and / or Jolt transformations. My flow file have multiple JSON records and as such attributes would take multiple values. And I don't want to split my records, this is really resource consuming.

Best regards,

Stéphane

[#5]

Hello@stephane_davy 

I'm sorry, I haven't seen that record-processing is a "must".

Because I haven't much experience with this kind of syntax I sadly will not be able to help you here.

But if you find a solution just be so kind and show it all of us here. Thank you!

[#6]

Hello @justenji 

Thanks anyway to have taken some time to look after my issue. For now, the only solution I have found is a groovy script.

Stéphane

[#7]

I need to test this, but actually it seems that the queryRecord is exactly why I need. It is possible to build some complex conditions using SQL-like language on the record content and then perform some routing based on that

[#8]

@stephane_davy 
I'm working in the same direction because I couldn't believe that there is no possibility.

But I'm still struggling with problems of the correct MergeRecord.
So we both go the same way.

[#9 · Accepted solution]

@stephane_davy 
Sorry I couldn't work on this further last week. Now my solution looks like this.

• JSON from GenerateFlowFile

[
      {
         "myfield": "JustForHavingJson",
         "myfield1": "A"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "B"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "C"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "D"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "E"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "X"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": ""
      }
   ]

• Definitions of ControllerServices and AvroSchemaRegistry
  ControllerServices and AvroSchemaRegistry
• Flow
  Flow

• Details of RecordProcessing-Processors
  Details RecordProcessing-Processors

• Result FF-Content after MergeRecord:

{"myfield":"JustForHavingJson","myfield1":"A","myfield2":"C"}
{"myfield":"JustForHavingJson","myfield1":"B","myfield2":"C"}
{"myfield":"JustForHavingJson","myfield1":"C","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"E","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"X","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"D","myfield2":"G"}

Personally I wouldn't do a MergeRecord at the end but go on with the three single connections of the UpdateRecord processors.

Do you think this could be a possible solution or have you found a better way to do this?

[#10]

Hello @justenji 

Thanks for the detailed explanation, this is also what I have tested on my side and in my opinion this does the job being "record oriented" 🙂

Have a nice day