How to Merge the original data with API response data based on the API response values and the array of json indexes in Apache NiFi

[#1]

Orginal data:
{
"data": [
{
"id": "1234569",
"Date": "2022-08-22"
},
{
"id": "1234567",
"Date": "2022-08-22"
},
{
"id": "34567",
"Date": "2022-08-22"
}
]
}

API Response:
[ {
"Record 2" : "Invalid values for emp info"
}, {
"Record 3" : "Invalid values for emp info"
} ]

Where Record 1,Record 2,Record 3 is array of json object index. Here we have three records in orginal data, Record 1 passed successfully and remaining two failed and due to that in response we are getting Record 2 and Record 3

i.e. if index of orginal data = value of Record (for example original data's index 1 is passed successfully so it should not merge. In orginal Response Index 2 failed, so it should merge with Record 2 and so on.. ) then final output should look like below:

{
"data": [
{
"id": "1234567",
"Date": "2022-08-22",
"Record 2" : "Invalid values for emp info"
},
{
"id": "34567",
"Date": "2022-08-22",
"Record 3" : "Invalid values for emp info"
}
]
}

How can I do that? Please help.

[#2 · Accepted solution]

Hi,

It seems like your best and out of the box option is to use the Fork Enrichment \ Join Enrichment processor. If each record in the original data has an equivalent record in the response then that should be super easy and you can use Wrapper or Insertion Strategy to enrich your data to something similar to what you expect, however since you mentioned that successful records wont have  an equivalent record in the response ,then you can take advantage of the SQL Enrichment strategy where you can do sql join between the two input (original & response) using common ID and generate the desired out. In Sql Strategy you have a lot of flexibility but you have to be careful with the performance. The challenge here  to use the SQL strategy or any Join Enrichment strategies is to have common link between the two, for your case the only link is the record index, so you might use Json Jolt to produce such link, For example your original data will look like this:

{
"data": [
{

"index" : "Record 1",
"id": "1234569",
"Date": "2022-08-22"
},
{

"index" : "Record 2",
"id": "1234567",
"Date": "2022-08-22"
},

...

and you response will  look like this after jolt transformation :

{

"Index":"Record 2",
"Record 2" : "Invalid values for emp info"
}, {

"Index":"Record 3"
"Record 3" : "Invalid values for emp info"
} ]

In this case you can do SQL join enrichment using "index" as the join key link

For more info regarding fork \join enrichment , see the following links:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apach...

Please, if you find this is helpful please accept solution.

Thanks

[#3]

Hi @SAMSAL ,

Okay, will try the solution by you. 

But the point is original data is travelling first and response will come after then how original response will wait? when i got the response from the api then only both data should merged

Can you please help 

[#4]

Hi,

Yes that is the point of the fork\join enrichment. the original flow file will wait until both response and original meet at the join enrichment.

[#5]

@SAMSAL 

Okay thanks, I am trying to generate the index of the array using jolt but the index always start from 0. How can get it from 1 so that the data will be in sync 

[#6]

can you send me sample input and your json jolt spec?

[#7]

@SAMSAL Thank you for your help.