Created 12-12-2018 07:20 PM
Hi,
i have an avro file that i want to merge with a different avro file,
the only difference between them is that one of the avro files have an additional column.
the columns are nullables and their default is set to null
for example, file1 schema:
{ "name" : "col1", "type" : ["null","string"], "default":null }
file2 schema:
{ "name" : "col1", "type" : ["null","string"], "default":null },{ "name" : "col2", "type" : ["null","string"], "default":null }
file1:
col1 billy chloe ethan
file2:
col1 col2 jhon null alex grey chad null
nifi is refusing to merge those files with the mergeContent and the mergeRecord processors. The first flowfile is being sent to relationship failure and second one to success.
i want that after the merge the file will look like this:
col1 col2 billy null chloe null ethan null jhon null alex grey chad null
does anybody have an idea or a solution how to get nifi to merge those files?
Thanks
Created 12-12-2018 09:51 PM
What's your Nifi version?
Could you be encountering this issue Issue with the MergeContent Processor when processing Avro files
Created 12-13-2018 02:54 AM
If you use "MergeRecord" processor then define RecordReader/Writer controller services avro schema with 2 columns (i.e. col1,col2).
Avro Schema:
{ "type": "record", "name": "path_sch", "fields": [ { "name": "col1", "type": ["null","string"]}, { "name": "col2", "type": ["null","string"]} ] }
I tried with same case as mentioned in the question and it worked as expectedly, Please find the attached templated here.
Created 12-13-2018 10:43 AM
@Geoffrey Shelton Okot
thanks for your reply, my nifi version is 1.7.0
Created 12-13-2018 10:46 AM
@Shu
Thank you for your answer,
my files are avro and not csv,
and they are very large so conversion to another format will take a lot of time