How to merge 2 flow file content coming from 2 different NiFi flow based on some matching condition ?

[#1]

Hi,

Can we merge 2 different JSON coming from 2 different flow files using any processor of NiFi. For example :

Input 1 :

{
"Table": "myTable",
"Database": "Database",
"Column": "Column1"
}

Input 2 :

{
"Table": "myTable",
"Database": "Database",
"Column": "Column2"
}

Now the challenge is how we can merge above 2 JSON based on matching condition of Table and Database and merge content of both JSON in a one JSON dictionary.

Output :

{
"Table": "myTable",
"Database": "Database",
"Column": [
"Column2",
"Column1"
]
}

Thank you in advance,

Subash

[#2 · Accepted solution]

This type of streaming join is not really what Apache NiFi was made to do...

You can do "look up" joins in NiFi where a processor has some kind of reference data set loaded in memory, and a flow file comes in and some value from the flow file is used to look up something in the reference data set and add that back into the data in the flow file, basically enrichment. There are some recent components added to help with this (LookupService and LookupRecord).

True streaming joins would better handled in a stream processing platform like Storm.

[#3]

Thank you @Bryan Bende, I will try Storm.