Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi: how to combile 14 000 flowfile in one file?

Solved Go to solution
Highlighted

Nifi: how to combile 14 000 flowfile in one file?

I have 1400 flowFile with same name and i want to enroll them into one file, i have read that i can use mergeContent processor for this purpose but when the amount of flowfile is more than 10000 it can't combine them in one file, what should i change to make this processor work properly? here is my configuration image example:

45519-mergeexmple.png

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Nifi: how to combile 14 000 flowfile in one file?

@sally sally

Check the back pressure object threshold value in the connection feeding the MergeContent processor to make sure it has been changed to a value that will allow enough files to queue up.

View solution in original post

2 REPLIES 2
Highlighted

Re: Nifi: how to combile 14 000 flowfile in one file?

What NiFI version are u using? You might be running into https://issues.apache.org/jira/browse/NIFI-516, which is already fixed. If you want to merge group of 1400 flowfiles into single file every time, then you should set the Minimum Number of entries as 1400. You can significantly lower your maximum number of Bins based on system resources. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache...

Highlighted

Re: Nifi: how to combile 14 000 flowfile in one file?

@sally sally

Check the back pressure object threshold value in the connection feeding the MergeContent processor to make sure it has been changed to a value that will allow enough files to queue up.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here