Support Questions

Find answers, ask questions, and share your expertise

Nifi: how to combile 14 000 flowfile in one file?

avatar
Contributor

I have 1400 flowFile with same name and i want to enroll them into one file, i have read that i can use mergeContent processor for this purpose but when the amount of flowfile is more than 10000 it can't combine them in one file, what should i change to make this processor work properly? here is my configuration image example:

45519-mergeexmple.png

1 ACCEPTED SOLUTION

avatar

@sally sally

Check the back pressure object threshold value in the connection feeding the MergeContent processor to make sure it has been changed to a value that will allow enough files to queue up.

View solution in original post

2 REPLIES 2

avatar

What NiFI version are u using? You might be running into https://issues.apache.org/jira/browse/NIFI-516, which is already fixed. If you want to merge group of 1400 flowfiles into single file every time, then you should set the Minimum Number of entries as 1400. You can significantly lower your maximum number of Bins based on system resources. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache...

avatar

@sally sally

Check the back pressure object threshold value in the connection feeding the MergeContent processor to make sure it has been changed to a value that will allow enough files to queue up.