Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

[Nifi] - Multiple predicate in recordpath filter

avatar
Contributor

Hello all,

 

I'd like to use the updateRecord processor to create some fields in my JSON. But I'd like to use some conditions to do that. In the recordpath documentation, I can see that there is a "filter" capability, which works like this: /field[filter]/path, but as far as I can see it's not possible to have multiple conditions in this filter. I'd like to do something like:

 

if (field1 == A or field1 == B) {

field2 = C

} else if (field1 == D and field1 != E) {
field2 = G
}

 

 

The example is silly but this is to give you the idea. By the way, except using a script, I don't know how to do that, except may be extracting the fields, put it as attribute and play with expression language but this would mean working with single-record flows and I don't want to do that.

 

Any idea?

 

Thanks for your suggestions

 

Stéphane

 

1 ACCEPTED SOLUTION

avatar
Master Collaborator

@stephane_davy 
Sorry I couldn't work on this further last week. Now my solution looks like this.

 

  • JSON from GenerateFlowFile
[
      {
         "myfield": "JustForHavingJson",
         "myfield1": "A"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "B"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "C"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "D"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "E"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "X"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": ""
      }
   ]

 

  • Definitions of ControllerServices and AvroSchemaRegistry
    ControllerServices and AvroSchemaRegistryControllerServices and AvroSchemaRegistry
  • Flow
    FlowFlow

 

  • Details of RecordProcessing-Processors
    Details RecordProcessing-ProcessorsDetails RecordProcessing-Processors

 

  • Result FF-Content after MergeRecord:
{"myfield":"JustForHavingJson","myfield1":"A","myfield2":"C"}
{"myfield":"JustForHavingJson","myfield1":"B","myfield2":"C"}
{"myfield":"JustForHavingJson","myfield1":"C","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"E","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"X","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"D","myfield2":"G"}


Personally I wouldn't do a MergeRecord at the end but go on with the three single connections of the UpdateRecord processors.

 

Do you think this could be a possible solution or have you found a better way to do this?

View solution in original post

9 REPLIES 9

avatar
Master Collaborator

@stephane_davy Maybe you want to give this solution a try?
Hope the explanation is clear. If not feel free to ask again.
<Picture removed>

avatar
Master Collaborator

@stephane_davy   Sorry, my above picture is a kind of vision test...

Flow and Step1 and 2Flow and Step1 and 2

Step 3 Rule 1Step 3 Rule 1Step 3 Rule 2Step 3 Rule 2Step 4Step 4

avatar
Contributor

Hello @justenji 

 

Thanks for you reply. The problem here is that you are working with attributes and expression language, which would mean to have one flowfile per json object. I'm trying to have a record-oriented flow-file, which means that everything happens with recordpath and / or Jolt transformations. My flow file have multiple JSON records and as such attributes would take multiple values. And I don't want to split my records, this is really resource consuming.

 

Best regards,

 

Stéphane

avatar
Master Collaborator

Hello@stephane_davy 

 

I'm sorry, I haven't seen that record-processing is a "must".

Because I haven't much experience with this kind of syntax I sadly will not be able to help you here.


But if you find a solution just be so kind and show it all of us here. Thank you!

avatar
Contributor

Hello @justenji 

 

Thanks anyway to have taken some time to look after my issue. For now, the only solution I have found is a groovy script.

Stéphane

avatar
Contributor

I need to test this, but actually it seems that the queryRecord is exactly why I need. It is possible to build some complex conditions using SQL-like language on the record content and then perform some routing based on that

avatar
Master Collaborator

@stephane_davy 
I'm working in the same direction because I couldn't believe that there is no possibility.

 

HCC_QueryRecord.PNGHCC_QueryRecord2.PNGBut I'm still struggling with problems of the correct MergeRecord.
So we both go the same way.

avatar
Master Collaborator

@stephane_davy 
Sorry I couldn't work on this further last week. Now my solution looks like this.

 

  • JSON from GenerateFlowFile
[
      {
         "myfield": "JustForHavingJson",
         "myfield1": "A"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "B"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "C"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "D"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "E"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": "X"
      },
      {
         "myfield": "JustForHavingJson",
         "myfield1": ""
      }
   ]

 

  • Definitions of ControllerServices and AvroSchemaRegistry
    ControllerServices and AvroSchemaRegistryControllerServices and AvroSchemaRegistry
  • Flow
    FlowFlow

 

  • Details of RecordProcessing-Processors
    Details RecordProcessing-ProcessorsDetails RecordProcessing-Processors

 

  • Result FF-Content after MergeRecord:
{"myfield":"JustForHavingJson","myfield1":"A","myfield2":"C"}
{"myfield":"JustForHavingJson","myfield1":"B","myfield2":"C"}
{"myfield":"JustForHavingJson","myfield1":"C","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"E","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"X","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"D","myfield2":"G"}


Personally I wouldn't do a MergeRecord at the end but go on with the three single connections of the UpdateRecord processors.

 

Do you think this could be a possible solution or have you found a better way to do this?

avatar
Contributor

Hello @justenji 

 

Thanks for the detailed explanation, this is also what I have tested on my side and in my opinion this does the job being "record oriented" 🙂

 

Have a nice day