Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to merge 2 flow file content coming from 2 different NiFi flow based on some matching condition ?

avatar
Expert Contributor

Hi,

Can we merge 2 different JSON coming from 2 different flow files using any processor of NiFi. For example :

Input 1 :

{
"Table": "myTable",
"Database": "Database",
"Column": "Column1"
}

Input 2 :

{
"Table": "myTable",
"Database": "Database",
"Column": "Column2"
}

Now the challenge is how we can merge above 2 JSON based on matching condition of Table and Database and merge content of both JSON in a one JSON dictionary.

Output :

{
"Table": "myTable",
"Database": "Database",
"Column": [
"Column2",
"Column1"
]
}

Thank you in advance,

Subash

1 ACCEPTED SOLUTION

avatar
Master Guru

This type of streaming join is not really what Apache NiFi was made to do...

You can do "look up" joins in NiFi where a processor has some kind of reference data set loaded in memory, and a flow file comes in and some value from the flow file is used to look up something in the reference data set and add that back into the data in the flow file, basically enrichment. There are some recent components added to help with this (LookupService and LookupRecord).

True streaming joins would better handled in a stream processing platform like Storm.

View solution in original post

2 REPLIES 2

avatar
Master Guru

This type of streaming join is not really what Apache NiFi was made to do...

You can do "look up" joins in NiFi where a processor has some kind of reference data set loaded in memory, and a flow file comes in and some value from the flow file is used to look up something in the reference data set and add that back into the data in the flow file, basically enrichment. There are some recent components added to help with this (LookupService and LookupRecord).

True streaming joins would better handled in a stream processing platform like Storm.

avatar
Expert Contributor

Thank you @Bryan Bende, I will try Storm.