question Re: How to merge multiple csv coming through flowfiles to single csv ? in Support Questions

How to merge multiple csv coming through flowfiles to single csv ?

vivek_b3ds — Sun, 18 Aug 2019 06:01:35 GMT

I am using apache nifi and retrieving data in bulk from mongodb in json format and converting to csv but the problem is multiple csv is generating for each json record. How could I merge all the csv in NIFI. I have tried MergeRecord processor but still multiple csv are generating. I not sure whether all the setting is valid for MergeRecord.

Re: How to merge multiple csv coming through flowfiles to single csv ?

ahadjidj — Thu, 29 Mar 2018 15:22:26 GMT

Hi @Vivek Singh

Have you tried by setting a blank "Correlation Attribute Name" ?

As you can see from the doc, this attribute is used to gather files having the same value in this attribute, so having the same filename which leads to the behavior your are seeing

If specified, two FlowFiles will be binned together only if they have the same value for this Attribute. If not specified, FlowFiles are bundled by the order in which they are pulled from the queue.

Re: How to merge multiple csv coming through flowfiles to single csv ?

vivek_b3ds — Thu, 29 Mar 2018 15:56:38 GMT

@Abdelkrim Hadjidj I tried. In this case it results in failure.

Re: How to merge multiple csv coming through flowfiles to single csv ?

ahadjidj — Thu, 29 Mar 2018 16:02:59 GMT

@Vivek Singh

when you say "multiple csv are generate" do you mean that no original csv is merged? you have X input flow files to MergeRecord and you get X out put? are they going through the success relation? I can see that you have flow files in the "original,failure", do you get errors ?

Re: How to merge multiple csv coming through flowfiles to single csv ?

vivek_b3ds — Sun, 18 Aug 2019 06:01:22 GMT

No I am not getting any error. It just hang and do nothing. I mean flow files are reaching to MergeProcessor after after converting record but after that nothing happens. You can see in the new image.

Re: How to merge multiple csv coming through flowfiles to single csv ?

ahadjidj — Thu, 29 Mar 2018 16:12:32 GMT

Ok so maybe you don't have enough flow files to create a new merged flow file. The decision to merge is based on two things : age of the bin and number of record. Do you have 1000 records going through the merge? if no try to set a short Max Bin Age to force the process to do the merge.

Re: How to merge multiple csv coming through flowfiles to single csv ?

vivek_b3ds — Thu, 29 Mar 2018 16:19:18 GMT

Thanks setting max bin age property works.