Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi: how to combile 14 000 flowfile in one file?

Solved Go to solution

Nifi: how to combile 14 000 flowfile in one file?

New Contributor

I have 1400 flowFile with same name and i want to enroll them into one file, i have read that i can use mergeContent processor for this purpose but when the amount of flowfile is more than 10000 it can't combine them in one file, what should i change to make this processor work properly? here is my configuration image example:

45519-mergeexmple.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Nifi: how to combile 14 000 flowfile in one file?

@sally sally

Check the back pressure object threshold value in the connection feeding the MergeContent processor to make sure it has been changed to a value that will allow enough files to queue up.

2 REPLIES 2
Highlighted

Re: Nifi: how to combile 14 000 flowfile in one file?

What NiFI version are u using? You might be running into https://issues.apache.org/jira/browse/NIFI-516, which is already fixed. If you want to merge group of 1400 flowfiles into single file every time, then you should set the Minimum Number of entries as 1400. You can significantly lower your maximum number of Bins based on system resources. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache...

Re: Nifi: how to combile 14 000 flowfile in one file?

@sally sally

Check the back pressure object threshold value in the connection feeding the MergeContent processor to make sure it has been changed to a value that will allow enough files to queue up.