Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to merge 2 flow file content coming from 2 different NiFi flow based on some matching condition ?

Solved Go to solution

How to merge 2 flow file content coming from 2 different NiFi flow based on some matching condition ?

New Contributor

Hi,

Can we merge 2 different JSON coming from 2 different flow files using any processor of NiFi. For example :

Input 1 :

{
"Table": "myTable",
"Database": "Database",
"Column": "Column1"
}

Input 2 :

{
"Table": "myTable",
"Database": "Database",
"Column": "Column2"
}

Now the challenge is how we can merge above 2 JSON based on matching condition of Table and Database and merge content of both JSON in a one JSON dictionary.

Output :

{
"Table": "myTable",
"Database": "Database",
"Column": [
"Column2",
"Column1"
]
}

Thank you in advance,

Subash

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to merge 2 flow file content coming from 2 different NiFi flow based on some matching condition ?

This type of streaming join is not really what Apache NiFi was made to do...

You can do "look up" joins in NiFi where a processor has some kind of reference data set loaded in memory, and a flow file comes in and some value from the flow file is used to look up something in the reference data set and add that back into the data in the flow file, basically enrichment. There are some recent components added to help with this (LookupService and LookupRecord).

True streaming joins would better handled in a stream processing platform like Storm.

2 REPLIES 2

Re: How to merge 2 flow file content coming from 2 different NiFi flow based on some matching condition ?

This type of streaming join is not really what Apache NiFi was made to do...

You can do "look up" joins in NiFi where a processor has some kind of reference data set loaded in memory, and a flow file comes in and some value from the flow file is used to look up something in the reference data set and add that back into the data in the flow file, basically enrichment. There are some recent components added to help with this (LookupService and LookupRecord).

True streaming joins would better handled in a stream processing platform like Storm.

Highlighted

Re: How to merge 2 flow file content coming from 2 different NiFi flow based on some matching condition ?

New Contributor

Thank you @Bryan Bende, I will try Storm.