NiFi - Need to help to merging/combine/join 2 Json FlowFile into 1

[#1]

I have 1 JSON FlowFile A (taken from 1 invoke HTTP processor) and 1 Json FlowFile B (also from 1 invoke HTTP processor). The problem I need to solve is to merge/combine/join them into 1 FlowFile C, whose schema is the sum of 2 Flow files A and B. I have tried using JoinEnrichment or MergeContent solution but with no success. Do you have any suggestions?

FlowFile A

{
  "id" : "CAT_c6848199-3594-453b-a733-c270bb837933",
  "categoryId" : "category-external-id-456",
  "updatedDate" : "2023-04-25T09:25:07.559Z",
  "title" : {
    "en-US" : "Industrial",
    "vi-VN" : "Research and Development"
  },
  "parentCategory" : null,
  "order" : 2,
  "description" : {
    "en-US" : "Aenean fermentum. Donec ut mauris eget massa tempor convallis. Nulla neque libero, convallis eget, eleifend luctus, ultricies eu, nibh.",
    "vi-VN" : "Nulla mollis molestie lorem. Quisque ut erat."
  },
  "mediaUrls" : [ "http://dummyimage.com/239x100.png/5fa2dd/ffffff", "http://dummyimage.com/104x100.png/ff4444/ffffff" ],
  "flexibleAttributes" : { },
  "seoTitle" : {
    "en-US" : "Books",
    "vi-VN" : "Support"
  },
  "seoKeywords" : {
    "en-US" : [ "augue", "lectus in quam fringilla rhoncus" ],
    "vi-VN" : [ "nec dui luctus", "nulla sed" ]
  },
  "seoDescription" : {
    "vi-VN" : "Aenean auctor gravida sem. Praesent id massa id nisl venenatis lacinia.",
    "en-US" : "Nulla neque libero, convallis eget, eleifend luctus, ultricies eu, nibh. Quisque id justo sit amet sapien dignissim vestibulum. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Nulla dapibus dolor vel est."
  },
  "updatedDateExternal" : null,
  "hashValue" : "e46e4c499f9a738c",
  "channelId" : "bfc12c9e-821d-4351-93db-4092508f5a6e"
}

Flowfile B

{
  "output_channels": [
    "24194622-e1b1-11ed-860f-325096b39f47",
    "241947f8-e1b1-11ed-847a-325096b39f47",
    "24194866-e1b1-11ed-98dc-325096b39f47",
    "241948c0-e1b1-11ed-9370-325096b39f47",
    "24194906-e1b1-11ed-95f0-325096b39f47",
    "2419494c-e1b1-11ed-a8f9-325096b39f47",
    "24194992-e1b1-11ed-8da8-325096b39f47",
    "241949ce-e1b1-11ed-9f22-325096b39f47",
    "24194a14-e1b1-11ed-a030-325096b39f47",
    "24194b9a-e1b1-11ed-bfd2-325096b39f47"
  ]
}

And the flowfile C I would like to have inserts field “output_channels” from Flow File B into a new field of FlowFile A

{
  "id" : "CAT_c6848199-3594-453b-a733-c270bb837933",
  "categoryId" : "category-external-id-456",
  "updatedDate" : "2023-04-25T09:25:07.559Z",
  "title" : {
    "en-US" : "Industrial",
    "vi-VN" : "Research and Development"
  },
  "parentCategory" : null,
  "order" : 2,
  "description" : {
    "en-US" : "Aenean fermentum. Donec ut mauris eget massa tempor convallis. Nulla neque libero, convallis eget, eleifend luctus, ultricies eu, nibh.",
    "vi-VN" : "Nulla mollis molestie lorem. Quisque ut erat."
  },
  "mediaUrls" : [ "http://dummyimage.com/239x100.png/5fa2dd/ffffff", "http://dummyimage.com/104x100.png/ff4444/ffffff" ],
  "flexibleAttributes" : { },
  "seoTitle" : {
    "en-US" : "Books",
    "vi-VN" : "Support"
  },
  "seoKeywords" : {
    "en-US" : [ "augue", "lectus in quam fringilla rhoncus" ],
    "vi-VN" : [ "nec dui luctus", "nulla sed" ]
  },
  "seoDescription" : {
    "vi-VN" : "Aenean auctor gravida sem. Praesent id massa id nisl venenatis lacinia.",
    "en-US" : "Nulla neque libero, convallis eget, eleifend luctus, ultricies eu, nibh. Quisque id justo sit amet sapien dignissim vestibulum. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Nulla dapibus dolor vel est."
  },
  "updatedDateExternal" : null,
  "hashValue" : "e46e4c499f9a738c",
  "channelId" : "bfc12c9e-821d-4351-93db-4092508f5a6e",
  "output_channels": [
    "24194622-e1b1-11ed-860f-325096b39f47",
    "241947f8-e1b1-11ed-847a-325096b39f47",
    "24194866-e1b1-11ed-98dc-325096b39f47",
    "241948c0-e1b1-11ed-9370-325096b39f47",
    "24194906-e1b1-11ed-95f0-325096b39f47",
    "2419494c-e1b1-11ed-a8f9-325096b39f47",
    "24194992-e1b1-11ed-8da8-325096b39f47",
    "241949ce-e1b1-11ed-9f22-325096b39f47",
    "24194a14-e1b1-11ed-a030-325096b39f47",
    "24194b9a-e1b1-11ed-bfd2-325096b39f47"
  ]
}

Thanks

[#2]

@ThienTrinh Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @cotopaul @SAMSAL @steven-matison  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

Hi

Not sure what Enrichment Strategy are you using but this can be accomplished using  Join Strategy "Insert Enrichment Fields" with Insertion Record Path "/":

Here is my data flow I'm using the GenerateFlowFile and the ReplaceText to simulate generating Flow File A & B respectively. 

The output of JoinEnrichment :

[ {
  "id" : "CAT_c6848199-3594-453b-a733-c270bb837933",
  "categoryId" : "category-external-id-456",
  "updatedDate" : "2023-04-25T09:25:07.559Z",
  "title" : {
    "en-US" : "Industrial",
    "vi-VN" : "Research and Development"
  },
  "parentCategory" : null,
  "order" : 2,
  "description" : {
    "en-US" : "Aenean fermentum. Donec ut mauris eget massa tempor convallis. Nulla neque libero, convallis eget, eleifend luctus, ultricies eu, nibh.",
    "vi-VN" : "Nulla mollis molestie lorem. Quisque ut erat."
  },
  "mediaUrls" : [ "http://dummyimage.com/239x100.png/5fa2dd/ffffff", "http://dummyimage.com/104x100.png/ff4444/ffffff" ],
  "flexibleAttributes" : { },
  "seoTitle" : {
    "en-US" : "Books",
    "vi-VN" : "Support"
  },
  "seoKeywords" : {
    "en-US" : [ "augue", "lectus in quam fringilla rhoncus" ],
    "vi-VN" : [ "nec dui luctus", "nulla sed" ]
  },
  "seoDescription" : {
    "vi-VN" : "Aenean auctor gravida sem. Praesent id massa id nisl venenatis lacinia.",
    "en-US" : "Nulla neque libero, convallis eget, eleifend luctus, ultricies eu, nibh. Quisque id justo sit amet sapien dignissim vestibulum. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Nulla dapibus dolor vel est."
  },
  "updatedDateExternal" : null,
  "hashValue" : "e46e4c499f9a738c",
  "channelId" : "bfc12c9e-821d-4351-93db-4092508f5a6e",
  "output_channels" : [ "24194622-e1b1-11ed-860f-325096b39f47", "241947f8-e1b1-11ed-847a-325096b39f47", "24194866-e1b1-11ed-98dc-325096b39f47", "241948c0-e1b1-11ed-9370-325096b39f47", "24194906-e1b1-11ed-95f0-325096b39f47", "2419494c-e1b1-11ed-a8f9-325096b39f47", "24194992-e1b1-11ed-8da8-325096b39f47", "241949ce-e1b1-11ed-9f22-325096b39f47", "24194a14-e1b1-11ed-a030-325096b39f47", "24194b9a-e1b1-11ed-bfd2-325096b39f47" ]
} ]

If that helps please accept solution.

Thanks

[#4]

@SAMSAL I did the same but it did not work as expected. do you have any special configuration for JsonTreeReader and JsonRecordSetWritter? The field “output_channel” was not inserted.

[#5]

Hi,

I'm not seeing any difference in the JsonTreeReader\Writer . I'm not sure why you are not getting the expected output. The only difference I see is that I'm using 1.16 version and you are on 1.20 so I'm not sure if the behavior is different between the two versions even though I suspect its the case. To troubleshoot and find out , try to create the same flow I created and use the GeneratedFlowFile and ReplaceText to generate the output A & B from your post above. If that works then I would double check your original flow and make sure that InvokeHttp is generating the desired output and the same response output is making it to the ForkEnrichment\JoinEnrichment as designed.

Let me know how that goes.

[#6]

@SAMSAL this is my template, could you help me try at your side,please? thanks
https://drive.google.com/file/d/1XOwQV1ujDhiSHvD_uBovYoyN1MzVMJBX/view?usp=sharing 

[#7]

I can confirm that there is behavior change in Fork\Join Enrichment processors between 1.16 and 1.20. I have uploaded the same template you sent using 1.16 and it worked using 1.16 JsonTreeReader\Writer , however the same template did not work on 1.20 using 1.20 processors and services. @MattWho, @steven-matison  Is this a bug or misunderstanding if how such processors and services should work in the later versions? Please advise. Thanks

[#8]

@ThienTrinh,

As workaround you can follow this scenario:

https://stackoverflow.com/questions/55118389/i-have-two-json-payload-i-want-to-merge-them-in-a-singl...