Skip to main content
sathish3389
sathish3389
Silver ⭐️⭐️
August 23, 2022
Solved

combining the Json data from two different processors in Apache Nifi

Hello,

I am working on one of the tasks in Nifi and I am stuck on of the issue, the issue is I am getting the Json data from jolt transforms processor and also I am getting the Json data from input port processor now the issue is I need to combine the data from two processor based on id’s.  How to combined the data from two processors based on common id ? Will you please help me out.

Thanks,

Sathish

SAMSAL
SAMSALAnswer
Platinum ⭐️⭐️⭐️
August 24, 2022

Hi,

Thanks for providing the diagram. Not sure if you can have both InvokeHttp in one group where one comes after the other instead of using two groups and input\output ports, but if you can then you can take advantage of the ForkEnrichment\JoinEnrichment processors (available in 1.16 and higher) using the following pattern:

 

SAMSAL_0-1661368176846.png

If both invokehttp output have the same order for the records then you can use the Wrapper or Insert Enrichment strategy, if the records are not in the same order you can use SQL strategy. For more information check the JoinEnrichment user manual :

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.17.0/org.apache.nifi.processors.standard.JoinEnrichment/additionalDetails.html

Hope that helps. If it does please accept solution.

View in thread

4 replies

SAMSAL
SAMSAL
Platinum ⭐️⭐️⭐️
August 23, 2022

Hi,

The full picture is not clear on how your flow works, for example where do you get the json that goes through the jolt processor and input port from? Is there a correlation between getting the json from the different sources where one json is generated before the other or the two json sources are independent of each other? If they are independent of each other then how do you know when to merge , for example what if the json from the input port did not come? Can you explain more on how your flow works and if there is a dependency where one json input drives the other or not?

Translate
sathish3389
sathish3389Author
Silver ⭐️⭐️
August 24, 2022

Hello,

The flow is using invoke http processor I will fetch the data from api and inother part the data we are pulling from another api sending the data to ouput port, now using input port I am pulling the data and there is common field which is id, using the id from both apis we need to merge the data that  is requirement, I will tell you the flow in diagrammatic way.

 

sathish3389_0-1661320157899.png

 

Translate
SAMSAL
SAMSALAnswer
Platinum ⭐️⭐️⭐️
August 24, 2022

Hi,

Thanks for providing the diagram. Not sure if you can have both InvokeHttp in one group where one comes after the other instead of using two groups and input\output ports, but if you can then you can take advantage of the ForkEnrichment\JoinEnrichment processors (available in 1.16 and higher) using the following pattern:

 

SAMSAL_0-1661368176846.png

If both invokehttp output have the same order for the records then you can use the Wrapper or Insert Enrichment strategy, if the records are not in the same order you can use SQL strategy. For more information check the JoinEnrichment user manual :

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.17.0/org.apache.nifi.processors.standard.JoinEnrichment/additionalDetails.html

Hope that helps. If it does please accept solution.

Translate
sathish3389
sathish3389Author
Silver ⭐️⭐️
August 27, 2022

Thanks for your help @SAMSAL 

Translate
Terms & ConditionsAccessibility statement