question Nifi: how to combile 14 000 flowfile in one file? in Archives of Support Questions (Read Only)

Nifi: how to combile 14 000 flowfile in one file?

salome_tkhilais — Sun, 18 Aug 2019 00:45:52 GMT

I have 1400 flowFile with same name and i want to enroll them into one file, i have read that i can use mergeContent processor for this purpose but when the amount of flowfile is more than 10000 it can't combine them in one file, what should i change to make this processor work properly? here is my configuration image example:

Re: Nifi: how to combile 14 000 flowfile in one file?

mpandit — Thu, 21 Dec 2017 22:17:18 GMT

What NiFI version are u using? You might be running into https://issues.apache.org/jira/browse/NIFI-516, which is already fixed. If you want to merge group of 1400 flowfiles into single file every time, then you should set the Minimum Number of entries as 1400. You can significantly lower your maximum number of Bins based on system resources. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.MergeContent/index.html

Re: Nifi: how to combile 14 000 flowfile in one file?

Wynner — Fri, 22 Dec 2017 04:20:26 GMT

@sally sally

Check the back pressure object threshold value in the connection feeding the MergeContent processor to make sure it has been changed to a value that will allow enough files to queue up.