Support Questions
Find answers, ask questions, and share your expertise

Need to confirm use of MergeContent is appropriate

Need to confirm use of MergeContent is appropriate

My flow is such

Get accounts from database

Get child records (type codes) for each account - one account can have multiple type codes

Derive attribute based on multiple child records

so if Acct A has one row with typecd = ABC set attribute ABC_IND =1

If Acct A has one row with typecd = XYZ set attribute XYZ_IND = 1

Merge all these attributes back to the original account record

Write the original and derived attributes to file

I am using the mergecontent to merge all the derived attributes flowfile back to the account flowfile based on attribute - acctNum

It worked on couple of records but when I ran for 8 records where 4 had child records and 4 did not I should have gotten 4 merged records written to file but I only got 2. My Merge Content is using bin-packing algorithm

Can merge be used for merging child records back to parent when I have not used any split processor?

12 REPLIES 12

Re: Need to confirm use of MergeContent is appropriate

This is how my flow looks like and my mergecontent configuration. What is happening is two records of different acctNums (ARN) are getting merged into one flowfile. Accroding the documentation this should not happen right.

9926-mergecontent-config.jpg

9925-merge-flow.jpg

Re: Need to confirm use of MergeContent is appropriate

I didn't understand question completely though, my guess is to use "RouteOnAttribute" before merge. In RouteOnAttribute, you can check if accountNumber of child and parent is same using expression language. If matched then merge.

Re: Need to confirm use of MergeContent is appropriate

Can you give me example based on the sample data I provided.

Re: Need to confirm use of MergeContent is appropriate

Rising Star

Please provide sample data, that reflects the data that flowed through the processors.

Re: Need to confirm use of MergeContent is appropriate

This is the sample data set

# ACCOUNT - PARENT {"ACCT_REF_NB": "1234", "ACCT_ORG_CD": 1111} {"ACCT_REF_NB": "5678", "ACCT_ORG_CD": 2222} {"ACCT_REF_NB": "2468", "ACCT_ORG_CD": 3333} {"ACCT_REF_NB": "1235", "ACCT_ORG_CD": 4444} {"ACCT_REF_NB": "1357", "ACCT_ORG_CD": 5555} {"ACCT_REF_NB": "4321", "ACCT_ORG_CD": 6666} {"ACCT_REF_NB": "8462", "ACCT_ORG_CD": 7777} {"ACCT_REF_NB": "9753", "ACCT_ORG_CD": 8888} # ACCOUNT DETAILS - CHILD {"ACCT_REF_NB": "5678", "TYPE_CD": "FPAY"} {"ACCT_REF_NB": "2468", "TYPE_CD": "FIIT"} {"ACCT_REF_NB": "2468", "TYPE_CD": "FPAY"} {"ACCT_REF_NB": "4321", "TYPE_CD": "FIIT"} {"ACCT_REF_NB": "9753", "TYPE_CD": "FPAY"} # ATTRIBUTE DERIVATION If an acocunt has a record with that type_cd then we need to derive the coresponding IND attribute to 1111 If child record for account contains type_cd = "FPAY" then set "FPAY_IND" = 1 If child record for account contains type_cd = "FIIT" then set "FIIT_IND" = 1 # DESIRED END RESULT AFTER DERIVATION AND MERGE {"ACCT_REF_NB": "1234", "ACCT_ORG_CD": 1111, "FPAY_IND": 0, "FIIT_IND": 0} {"ACCT_REF_NB": "5678", "ACCT_ORG_CD": 2222, "FPAY_IND": 1, "FIIT_IND": 0} {"ACCT_REF_NB": "2468", "ACCT_ORG_CD": 3333, "FPAY_IND": 1, "FIIT_IND": 1} {"ACCT_REF_NB": "1235", "ACCT_ORG_CD": 4444, "FPAY_IND": 0, "FIIT_IND": 0} {"ACCT_REF_NB": "1357", "ACCT_ORG_CD": 5555, "FPAY_IND": 0, "FIIT_IND": 0} {"ACCT_REF_NB": "4321", "ACCT_ORG_CD": 6666, "FPAY_IND": 0, "FIIT_IND": 1} {"ACCT_REF_NB": "8462", "ACCT_ORG_CD": 7777, "FPAY_IND": 0, "FIIT_IND": 0} {"ACCT_REF_NB": "9753", "ACCT_ORG_CD": 8888, "FPAY_IND": 1, "FIIT_IND": 0}

Re: Need to confirm use of MergeContent is appropriate

Super Collaborator

Hello, I guess it could be something very simple that we all have over looked. I just tried out the below configuration for my merge content. I have 10 messages each falling into 2 categories. The actype is my Correlation attribute and it takes only 2 value either a1 or a2. All files with a1 are merged into one flowfile and all with a2 to another. The bin age is set to 1 minute. Is it possible for us to debug your flow with your data set?

9918-merge-content-conf.png

9919-merge-content-op.png

Re: Need to confirm use of MergeContent is appropriate

This is how I expected it to work. The thing is I have done debug by running one step at a time with 4 records - 2 that have child and 2 that do not. It works when I do it one at a time with few records. But when I run the whole pipeline with more records it gives the weird behavior. In fact when I run with 8 records 4 of them being the same that I ran individually, even those previous records behave differently. This is just with trying to merge the child records. When I try to merge the child 2 of them work but remaining ones with different arn get dumped into 1 flow file.

There are 2 differences with your config and mine- You have selected Merge Format:Binary concatenation and Bin Age:60 secs. Do you think the merge format makes a difference. I would like ot keep it as a Flow File.

Re: Need to confirm use of MergeContent is appropriate

Super Collaborator

Merge Format does not make a difference. I had tried this before with flow file v3. The results are pretty much the same.

Re: Need to confirm use of MergeContent is appropriate

You were right. It was a small thing. I was specifying attribute name as ${ARN} whereas you had specified the attribute name directly so I changed to ARN and it stopped the weird behavior. But the issue I am facing now is I am getting 2 flow files for the parents that have child records one with my derived attribute set and one without.

So I have 8 parent records for which 4 have child records. Now I am getting 12 records after the merge. It looks like it is not merging parent and child records.

I do not know if it has to do with minimum number of entries being 1. But I cannot change minimum to 2 as those parent records that have no child records will have only 1 entry with that ARN value. Is there anything else that I should be changing.