Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to merge multiple csv coming through flowfiles to single csv ?

Solved Go to solution
Highlighted

How to merge multiple csv coming through flowfiles to single csv ?

Explorer

I am using apache nifi and retrieving data in bulk from mongodb in json format and converting to csv but the problem is multiple csv is generating for each json record. How could I merge all the csv in NIFI. I have tried MergeRecord processor but still multiple csv are generating. I not sure whether all the setting is valid for MergeRecord.

67428-setting.png

67427-client.png

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to merge multiple csv coming through flowfiles to single csv ?

Hi @Vivek Singh

Have you tried by setting a blank "Correlation Attribute Name" ?

As you can see from the doc, this attribute is used to gather files having the same value in this attribute, so having the same filename which leads to the behavior your are seeing

If specified, two FlowFiles will be binned together only if they have the same value for this Attribute. If not specified, FlowFiles are bundled by the order in which they are pulled from the queue.

View solution in original post

6 REPLIES 6
Highlighted

Re: How to merge multiple csv coming through flowfiles to single csv ?

Hi @Vivek Singh

Have you tried by setting a blank "Correlation Attribute Name" ?

As you can see from the doc, this attribute is used to gather files having the same value in this attribute, so having the same filename which leads to the behavior your are seeing

If specified, two FlowFiles will be binned together only if they have the same value for this Attribute. If not specified, FlowFiles are bundled by the order in which they are pulled from the queue.

View solution in original post

Highlighted

Re: How to merge multiple csv coming through flowfiles to single csv ?

Explorer

@Abdelkrim Hadjidj I tried. In this case it results in failure.

Highlighted

Re: How to merge multiple csv coming through flowfiles to single csv ?

@Vivek Singh

when you say "multiple csv are generate" do you mean that no original csv is merged? you have X input flow files to MergeRecord and you get X out put? are they going through the success relation? I can see that you have flow files in the "original,failure", do you get errors ?

Re: How to merge multiple csv coming through flowfiles to single csv ?

Explorer

No I am not getting any error. It just hang and do nothing. I mean flow files are reaching to MergeProcessor after after converting record but after that nothing happens. You can see in the new image.

67429-merge.png

Highlighted

Re: How to merge multiple csv coming through flowfiles to single csv ?

Ok so maybe you don't have enough flow files to create a new merged flow file. The decision to merge is based on two things : age of the bin and number of record. Do you have 1000 records going through the merge? if no try to set a short Max Bin Age to force the process to do the merge.

Highlighted

Re: How to merge multiple csv coming through flowfiles to single csv ?

Explorer

Thanks setting max bin age property works.

Don't have an account?
Coming from Hortonworks? Activate your account here