Compare data within the JSON using NIFI

Question

i have a JSON payload which contains latest and old orders. The data should be compared and output New, update, deleted. How can do it do that in NIFI?.

"latest_orders": [

{ "order_id": "1001", "customer": "Alice", "amount": 250 },

{ "order_id": "1002", "customer": "Bob", "amount": 300 },

{ "order_id": "1004", "customer": "Diana", "amount": 150 }

],

"old_orders": [

{ "order_id": "1001", "customer": "Alice", "amount": 200 },

{ "order_id": "1002", "customer": "Bob", "amount": 300 },

{ "order_id": "1003", "customer": "Charlie", "amount": 400 }

]

}

Output:

{ "order_id": "1001", "customer": "Alice", "amount": 250, "Action":"UPDATE" },

{ "order_id": "1002", "customer": "Bob", "amount": 300 ,"Action":"UPDATE"},

{ "order_id": "1004", "customer": "Diana", "amount": 150,"Action":"NEW" },

{ "order_id": "1003", "customer": "Charlie", "amount": 400,"Action":"DELETE" }

mburgess · Accepted Answer

You can use ExecuteGroovyScript with the following script:

def ff = session.get()
if (!ff) return

def obj = new groovy.json.JsonSlurper().parse(ff.read())
def outObj = []

// Find updated records
def old_ids = obj.old_orders.collect {it.order_id}
def latest_ids = obj.latest_orders.collect {it.order_id}
old_ids.intersect(latest_ids).each {order_id ->
def update_order = obj.latest_orders.find {it.order_id == order_id}
update_order.Action = 'UPDATE'
outObj += update_order
}

// Find deleted records
(old_ids - latest_ids).each {order_id ->
def delete_order = obj.old_orders.find {it.order_id == order_id}
delete_order.Action = 'DELETE'
outObj += delete_order
}

// Find new records
(latest_ids - old_ids).each {order_id ->
def new_order = obj.latest_orders.find {it.order_id == order_id}
new_order.Action = 'NEW'
outObj += new_order
}
ff.write('UTF-8', groovy.json.JsonOutput.toJson(outObj))
REL_SUCCESS << ff

DianaTorres · Answer

Hi @steven-matison @MattWho @mburgess Do you have any insights here? Thanks!

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.