Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

nifi merge avro with different schemas

avatar
Explorer

Hi,
i have an avro file that i want to merge with a different avro file,
the only difference between them is that one of the avro files have an additional column.
the columns are nullables and their default is set to null
for example, file1 schema:

{
	"name" : "col1",
        "type" : ["null","string"],
        "default":null
}

file2 schema:

{
	"name" : "col1",
        "type" : ["null","string"],
        "default":null
},{
	"name" : "col2",
        "type" : ["null","string"],
        "default":null
}

file1:

col1
billy
chloe
ethan

file2:

col1	col2
jhon	null
alex	grey
chad	null

nifi is refusing to merge those files with the mergeContent and the mergeRecord processors. The first flowfile is being sent to relationship failure and second one to success.
i want that after the merge the file will look like this:

col1	col2
billy	null
chloe	null
ethan	null
jhon	null
alex	grey
chad	null

does anybody have an idea or a solution how to get nifi to merge those files?
Thanks

4 REPLIES 4

avatar
Master Mentor

@Lior Sela

What's your Nifi version?

Could you be encountering this issue Issue with the MergeContent Processor when processing Avro files

avatar
Master Guru

@Lior Sela

If you use "MergeRecord" processor then define RecordReader/Writer controller services avro schema with 2 columns (i.e. col1,col2).

Avro Schema:

{
  "type": "record",
  "name": "path_sch",
  "fields":
  [
    { "name": "col1", "type": ["null","string"]},
    { "name": "col2", "type": ["null","string"]}
]
}

I tried with same case as mentioned in the question and it worked as expectedly, Please find the attached templated here.

mergerecord-avro.xml

avatar
Explorer

@Geoffrey Shelton Okot
thanks for your reply, my nifi version is 1.7.0

avatar
Explorer

@Shu
Thank you for your answer,
my files are avro and not csv,
and they are very large so conversion to another format will take a lot of time