Support Questions
Find answers, ask questions, and share your expertise

how to remove dupilcate values from nested json file using nifi

how to remove dupilcate values from nested json file using nifi

Explorer

Hi

I am trying to delete the duplicate fields and values from nested json file but unable to get through it .Please help me for any thoughts for same

here are the sample file and process followed for this challenge

input json file

{"_id":"AAA1","casenum":[{"casenum":"243","stotal":936},{"casenum":"238","stotal":1074},{"casenum":"243","stotal":936}],"total":210026}

output json file

{"_id":"AAA1","casenum":[{"casenum":"243","stotal":936},{"casenum":"238","stotal":1074}],"total":210026}

here want to delete dupillicate case key and value pair ({"casenum":"243","stotal":936}) , these the sameple part , nested json file contains lot of duplicae key values pairs are avaialbel

Process followed

1)Jolttrasnformjson :

[ { "operation": "shift", "spec": { "_id": "projectid", "caseid": { "*": "&" } } } ]

The jolt transfomjson not successfull due to _id field

2) followed runaggregationmongo processor with below aggregate query . but it was not working as expected result

[ {"$unwind": "$casenum"}, {"$project": {"casenum": "$casenum.caseid", "subtotal": "$casenum.stotal" }} ]

Please provide the necessary script which will help me . Thanks in advance