Support Questions

Find answers, ask questions, and share your expertise

How to Merge files together by file attribute in Nifi.?

avatar
Super Collaborator

Hi ,I have 1000 files in a folder , file names have date(yyyymmdd) in it. for ex data_20160810.csv...i have 200 files per each day (so 5 days files). I want merge those files by date . so if successful, i my output folder will contain 5 merged files one for each day.

i am trying to do this by "Correlation Attribute Name" property..but still it is not merging in groups.

what am i doing wrong.?

6527-mergebydate.png

1 ACCEPTED SOLUTION

avatar

@Saikrishna Tarapareddy

The attribute to correlate on needs to be present in the flowfile for the Merge processor to use it. If you are using FetchFile to get the file, you can add an attribute into that processor using the filename or the substring of the file name. Then it will be present in the flowfile for subsequent processors to use.

View solution in original post

11 REPLIES 11

avatar
New Contributor

Using the below configuration

93759-update-attribute.jpg

93760-mergecontent.jpg

avatar
Super Collaborator

@Bryan Bende ,

RouteOnAttribute will know the value of header.? Don't i have to read the value of first line to know the header.? if i have to read first line then should i use SplitText which i read creates a flow file for each line which i want to avoid as i am trying to move thousands of files . below is a simple flow ..i created 5 files , 4 with header foo and one with a different header. i want to merge all the 4 files with foo as header. tried the approach suggested by you above , but its routing everything to unmatched.

6984-filefilter.png

6985-filefilter2.png