About mergecontent question

[#1]

My setup looks like above.

My question is: what if I have 11111 in total, I think each 1000 flowfiles will be merge, how about the leftover 111 flowfiles? how can I deal with the leftover flowfiles?

Thanks!

[#2 · Accepted solution]

@jarviszzzz , well the remaining 111 flowfiles will remain in the queue until the minimum number of entries is reached again. 

As a best practice, confirmed by the documentation as well, it is recommended to use the property Max Bin Age as well so that you could avoid the behavior your just raised in this post :).

Max Bin Age specifies the maximum amount of time that your flow can wait until the flowfiles are merged .... this basically provides a means to set a timeout on your bin so that even if you are no longer receiving any data, the flowfiles won't remain stuck in your queue. For this property you can use any positive integer with the desired time unit (seconds, minutes, hours) 

In your specific case, if you are going to set the Max Bin Age to 60 minutes, you will generate 10 bins counting 1000 entries and after 60 minutes, if no other data comes in your queue, you will generate a new bin, containing only the remaining 111 entries. 

More details can be found here, where the logic of this processes is quite well described:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.23.2/org.apach... 

[#3]

@cotopaul Dear Cotopaul, Thank you so much for helping!!