Jolt transformation Array to Flatten

[#1]

Hi Jolt Experts,

Iam looking to use the Jolt transformation in NIFI and looking for a help.

{
"header": {
"timeStamp": "1681441547144",
"activityId": "5142917973836294867"
},
"resource": {
"drniId": "261968731599587823",
"@type": "Order",
"subtype": "Retail",
"name": "CCCCC",
"resourceCharacteristic": [
{
"propname": "aaa",
"value": "P_1655442"
},
{
"name": "latest",
"value": [
11,
200,
300,
16,
2,
3
]
}]
}
}

To have an output like

{
"records" : [ {
"timeStamp" : "1681441547144",
"activityId" : "5142917973836294867",
"drniId" : "261968731599587823",
"type" : "Order",
"subtype" : "Retail",
"name" : "CCCCC",
"propname" : "aaa",
"value" : "P_1655442"

}, {
"timeStamp" : "1681441547144",
"activityId" : "5142917973836294867",
"drniId" : "261968731599587823",
"type" : "Order",
"subtype" : "Retail",
"name" : "CCCCC",
"propname" : "latest",
"value" : "11,200,300,16,2,3"
} ]
}

Basically copying all the top header and resource into to new by flattening it. Any help appreciated.

[#2 · Accepted solution]

Hi,

Try the following spec:

[
  {
    // This is to remove the @ character from @type property because I could not
    // figure out how to make it work in the second shift using escape character '\'
    "operation": "shift",
    "spec": {
      "header": "&",
      "resource": {
        "\\@type": "resource.type",
        "*": "resource.&"
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "resource": {
        "resourceCharacteristic": {
          "*": {
            "name": "records[&1].propname",
            "*": "records[&1].&",
            "@(2,name)": "records[&1].name",
            "@(2,type)": "records[&1].type",
            "@(2,subtype)": "records[&1].subtype",
            "@(2,drniId)": "records[&1].drniId",
            "@(3,header.activityId)": "records[&1].activityId",
            "@(3,header.timeStamp)": "records[&1].timeStamp"
          }
        }
      }
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "records": {
        "*": {
          "value": "=join(',', @(1,value))"
        }
      }
    }
  }
]

If that helps please accept solution.

Thanks

[#3 · Accepted solution]

Hi,

Try the following spec:

[
  {
    // This is to remove the @ character from @type property because I could not
    // figure out how to make it work in the second shift using escape character '\'
    "operation": "shift",
    "spec": {
      "header": "&",
      "resource": {
        "\\@type": "resource.type",
        "*": "resource.&"
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "resource": {
        "resourceCharacteristic": {
          "*": {
            "name": "records[&1].propname",
            "*": "records[&1].&",
            "@(2,name)": "records[&1].name",
            "@(2,type)": "records[&1].type",
            "@(2,subtype)": "records[&1].subtype",
            "@(2,drniId)": "records[&1].drniId",
            "@(3,header.activityId)": "records[&1].activityId",
            "@(3,header.timeStamp)": "records[&1].timeStamp"
          }
        }
      }
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "records": {
        "*": {
          "value": "=join(',', @(1,value))"
        }
      }
    }
  }
]

If that helps please accept solution.

Thanks

[#4]

Thanks @SAMSAL .It worked. In the same way how do i get an simillar output for "resourceRelationshipCharacteristic" . i tried few option but not getting the desirable output.

{
"header": {
"version": "1.0",
"timeStamp": "1681441547144",
"activityId": "5142917973836294867",
"action": "modify"
},
"resource": {
"drniId": "261968731599587823",
"@type": "VVVVV",
"subtype": "BBBBBB",
"name": "VVVVVVV",
"resourceCharacteristic": [
{
"name": "matchingkey",
"value": "CCCCC",
"@type": "string"
}
],
"resourceRelationship": [
{
"drniId": "261968731599587635",
"@type": "CCCCCC",
"subtype": "OTU4",
"name": "BBBBBB",
"relationDrniId": "5143209344417655493",
"resourceRelationshipCharacteristic": [
{
"name": "p3Latest",
"value": true,
"@type": "boolean"
},
{
"name": "isNotInRoute",
"value": "N",
"@type": "string"
}
]
},
{
"drniId": "261988866406271344",
"@type": "BVDFDFS",
"subtype": "VDFDG",
"name": "ABCC",
"relationDrniId": "5143209344417655494",
"resourceRelationshipCharacteristic": [
{
"name": "p3Latest",
"value": true,
"@type": "boolean"
},
{
"name": "isNotInRoute",
"value": "N",
"@type": "string"
}
]
}
]
}
}

[#5]

Hi ,

Can you provide the expected json output? The json you provided seems complex and nested. I'm not sure what exactly you are expecting.

[#6]

Basically inhertied properties of header and resource inside resourceRelationshipCharacteristic

{
"records" : [ {
"relationDrniId" : "5143209344417655493",
"timeStamp" : "1681441547144",
"activityId" : "5142917973836294867",
"action" : "modify",
"name": "p3Latest",
"value": true,
"@type": "boolean"

},

{
"relationDrniId" : "5143209344417655493",
"timeStamp" : "1681441547144",
"activityId" : "5142917973836294867",
"action" : "modify",
"name": "isNotInRoute",
"value": "N",
"@type": "string"

} ,




{
"relationDrniId" : "5143209344417655494",
"timeStamp" : "1681441547144",
"activityId" : "5142917973836294867",
"action" : "modify",
"name": "p3Latest",
"value": true,
"@type": "boolean"

} ,


"relationDrniId" : "5143209344417655494",
"timeStamp" : "1681441547144",
"activityId" : "5142917973836294867",
"action" : "modify",
"name": "isNotInRoute",
"value": "N",
"@type": "string"

}



]
}

[#7 · Accepted solution]

Hi,

This one was a little trickier from the first post, but it seems that there is nothing that you cant do with Jolt 🙂 . Please try the following spec:

[
  {
    // combine all resourceRelationshipCharacteristic under one group
    // and assign each element under the group unique key depending on
    // its index location starting from first array under resourceRelationship (&3) and
    // and ending with nested array resourceRelationshipCharacteristic ($1) so
    // each element will have unique name 00,01,10,11...
    "operation": "shift",
    "spec": {
      "resource": {
        "resourceRelationship": {
          "*": {
            "resourceRelationshipCharacteristic": {
              "*": {
                "@(6,header.action)": "&3&1.action",
                "@(6,header.timeStamp)": "&3&1.timeStamp",
                "@(2,relationDrniId)": "&3&1.relationDrniId",
                "*": "&3&1.&"
              }
            }
          }
        }
      }
    }
  },
  {
    // bucket each element (00,01,10,11) value into new Array records
    "operation": "shift",
    "spec": {
      "*": "records.[#1]"
    }
  }
]

Hope that helps.

I wonder if there is better\cleaner way @araujo  @cotopaul @steven-matison 

[#8]

Thanks @SAMSAL You are a genius. I was caught with the duplicate index array and couldn't resolve. 

[#9]

I have used the below spec for moving some of the resource characterstic to top level but iam getting null values. 

[
{
"operation": "shift",
"spec": {
"header": {
"timeStamp": "records.inv_activity_ts",
"activityId": "records.inv_activity_id",
"action": "records.action"
},
"resource": {
"drniId": "records.inv_id",
"subtype": "records.inv_subtype",
"name": "records.inv_name",
"resourceCharacteristic": {
"*": {
"name": {
"matchingkey": "records.matchingkey-value",
"status": "records.status-value"
},
"value": {
"matchingkey": "records.matchingkey-value",
"status": "records.status-value"
}
}
}
}
}
}
]

Output iam getting

{
"records" : {
"inv_activity_ts" : "1670484663189",
"inv_activity_id" : "256388257993155783",
"action" : "create",
"inv_id" : "256383859946641699",
"inv_subtype" : "Backplane Connection",
"inv_name" : "Backplane Connection",
"matchingkey-value" : null,
"status-value" : null
}
}

What is mistake iam making?.