Support Questions
Find answers, ask questions, and share your expertise

Merge two flowfiles into one stream using apache-nifi

Explorer

I have following scenario with apache-nifi

Flowfile1 :

 

id,name
1,sachith
2,nalaka
3,muha

 

 

Flowfile2:

 

 

id,email
1,sachith@email.com
2,nalaka@email.com

 

 

 

After merging these two flowfiles, result should be :

 

 

id,name,email
1,sachith,sachith@email.com
2,nalaka,nalaka@email.com
3,muha,

 

 

 

Here most important part is that, 3,muha, row should be there with email field empty.

I have refered Merge two JSON flowfile together in NiFi, which is very similar to my problem. But not with adding empty value for the row.

How to join two CSVs with Apache Nifi

For now I have tried with MergeContent processor, but it appends two flowfiles. Not adding as new columns with unique id.

Very similar question here,Merge two schemas into one in Apache nifi, but it doesn't have much details.

 

1 REPLY 1

Super Collaborator

The reason why you are struggling to find a solution is that your challenge is not something we would call a merge. To do this properly we would need a 'join'.

 

As mentioned, this is not a typical thing that you want to do with Nifi. Maybe you can find a workaround by using merge creativity, but the real solution is to use a streaming engine that is capable of streaming joins, like Spark. 


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'. Also check out my techincal portfolio at https://portfolio.jaheruddin.nl