question Re: How to create only one file by MergeContent processor on the cluster in Archives of Support Questions (Read Only)

How to create only one file by MergeContent processor on the cluster

amillion8 — Wed, 27 Dec 2017 01:57:07 GMT

I need to extract data from a relational database and load it into S3 bucket. I have a 5 node cluster, and use "GenerateTableFetch" (Primary node) --> "ExecuteSQL" (All nodes) combination to read the data in parallel. I also need to merge extracted data into a single file before loading it into S3, but my "MergeContent" processor produces multiple files in S3. Is there a way to get this done? The full flow looks like this:

"GenerateTableFetch" --> "ExecuteSQL" --> "MergeConent" --> "ConvertAvroToJSON" --> "UpdateAttribute" --> "CompressContent" --> "PutS3Object"

Re: How to create only one file by MergeContent processor on the cluster

Shu_ashu — Wed, 27 Dec 2017 09:01:53 GMT

@Alex M

Can you please share more details of your configurations of MergeContent Processor.

Refer to below community links How to configure Merge Content processor.

https://community.hortonworks.com/questions/148294/nifi-problems-with-emply-queue.html?childToView=148310#comment-148310

https://community.hortonworks.com/questions/149047/nifi-how-to-handle-with-mergecontent-processor.html?childToView=148309#answer-148309

Re: How to create only one file by MergeContent processor on the cluster

amillion8 — Wed, 27 Dec 2017 09:38:16 GMT

Attached is the template.

Thanks,

Alex

Re: How to create only one file by MergeContent processor on the cluster

TimothySpann — Wed, 27 Dec 2017 09:47:06 GMT

1. Change the amount and delay of the merge.

2. You can add an Enforce Order processor (only one primary node)

3. Make all connections FirstInFirstOutPrioritizer

Re: How to create only one file by MergeContent processor on the cluster

Shu_ashu — Sun, 18 Aug 2019 09:46:31 GMT

@Alex M,

You need to change Minimum Group Size as per your requirement like (1 B,1 KB,1 MB,1 GB..)

Example:-

As you can see below configs i changed Minimum Group Size as 10 MB //The minimum size of for the bundle.

let's consider your each flow files size is 1 MB each so the processor will wait until the group size reaches to 10 MB and then bundles all the flowfiles as 1(i.e 10 flowfiles merged as 1 flowfile after merge content processor).

if the flowfiles won't meet the minimum group size requirement then the flowfiles are going to wait before merge content processor until it reaches the minimum group size.

How to force merge flowfiles?

By specifying Max Bin Age property

No matter how many Flowfiles have been assigned to a given bin, that bin will be merged once the bin has existed for this amount of time.

let's consider if i set Max Bin Age property to 10 min and i had only 5 flowfiles having 5 MB over all queue size before merge content processor and our minimum group size property is 10 MB.

The queue will never meet the minimum group size requirement that means flowfiles will be queued for ever there to over come this situation we have added 10 min as max bin age so once the flowfile been in the queue for 10 min then the processor going to merge the flowfiles although they haven't meet the minimum group size requirement also.

About all the other properties in Merge Content processor please refer to the links that i mentioned above answer.

Let me know if you are having any questions..!!

Re: How to create only one file by MergeContent processor on the cluster

amillion8 — Wed, 27 Dec 2017 22:38:14 GMT

@Shu

Thank you - after some tweaking and tuning of the parameters you mentioned I was able to achieve desired results.

Alex

Re: How to create only one file by MergeContent processor on the cluster

SS_Jin — Fri, 06 Dec 2024 13:34:31 GMT

I was having similar issue, where I had around 7000+ flow files and I needed to merge them into one. Provided that, I didn't know what will be the exact number of flow files, I couldn't specify that number in Minimum Number of Entries attribute of mergeContent processor to make it only 1 flow file after merge.

After trying a lot of things, including trying to count the flow files or induce a delay before sending them off to merge, I finally found the solution on the mergeContent documentation below :

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.MergeContent/index.html#:~:text=The%20%27Defragment%27%20algorithm%20combines%20fragments%20that%20are%20associated%20by%20attributes%20back%20into%20a%20single%20cohesive%20FlowFile.%20The%20%27Bin%2DPacking%20Algorithm%27%20generates%20a%20FlowFile%20populated%20by%20arbitrarily%20chosen%20FlowFiles

And there you go ! setting 'Merge Strategy' property to 'Defragment' and 'Attribute Strategy' property to 'Keep Only Common Attributes' fixes the problem for me.

In all my flow files, they have common attributes and are now merged into a single flow file.

The original question is from 2017, but I hope it helps out somebody else looking for an answer.

You can check the original answer here https://stackoverflow.com/questions/56356164/nifi-merging-all-of-necessary-flowfile-in-one-shot-with-mergecontent-processor/79257994#79257994