How to merge multiple csv coming through flowfiles to single csv ?

[#1]

I am using apache nifi and retrieving data in bulk from mongodb in json format and converting to csv but the problem is multiple csv is generating for each json record. How could I merge all the csv in NIFI. I have tried MergeRecord processor but still multiple csv are generating. I not sure whether all the setting is valid for MergeRecord.

[#2 · Accepted solution]

Hi @Vivek Singh

Have you tried by setting a blank "Correlation Attribute Name" ?

As you can see from the doc, this attribute is used to gather files having the same value in this attribute, so having the same filename which leads to the behavior your are seeing

If specified, two FlowFiles will be binned together only if they have the same value for this Attribute. If not specified, FlowFiles are bundled by the order in which they are pulled from the queue.

[#3]

@Abdelkrim Hadjidj I tried. In this case it results in failure.

[#4]

@Vivek Singh

when you say "multiple csv are generate" do you mean that no original csv is merged? you have X input flow files to MergeRecord and you get X out put? are they going through the success relation? I can see that you have flow files in the "original,failure", do you get errors ?

[#5]

No I am not getting any error. It just hang and do nothing. I mean flow files are reaching to MergeProcessor after after converting record but after that nothing happens. You can see in the new image.

[#6]

Ok so maybe you don't have enough flow files to create a new merged flow file. The decision to merge is based on two things : age of the bin and number of record. Do you have 1000 records going through the merge? if no try to set a short Max Bin Age to force the process to do the merge.

[#7]

Thanks setting max bin age property works.