problem with joltTransform-record out of place when null

[#1]

Hi guys! I have some problem with joltTransformation.I have json like that:

{
"system": "TGG",
"income": "1000.0",
"costs": "700.0",
"id": "1",
"dateOfEvent": 1705446000000,
"name": "JAMES",
"surname": "SMITH",
"mail": "jsmith@gmail.com",
"dateOfBirth": 1705446000000,
"country": "US",
"city": "ORLANDO",
"eventId": "22",
"eventName": "EVENT",
"personalData": "true"
}

And I need output like this:

{
"system" : "TGG",
"income" : "1000.0",
"costs" : 700.0,
"id" : "1",
"dateOfEvent" : 1705446000000,
"person" : {
"name" : "JAMES",
"surname" : "SMITH",
"mail" : "jsmith@gmail.com",
"dateOfBirth" : 1705446000000,
"address" : {
"country" : "US",
"city" : "ORLANDO"
}
},
"eventData" : {
"personalData" : true,
"eventName" : "EVENT",
"eventId" : "22"
}
}

I have JoltSpec like this:

[

{
"operation": "shift",
"spec": {
"system": "system",
"income": {
"": {
"": "income"
},
"*": {
"$": "income"
}
},
"costs": {
"": {
"": "costs"
},
"*": {
"$": "costs"
}
},
"id": "id",
"dateOfEvent": {
"": {
"": "dateOfEvent"
},
"*": {
"$": "dateOfEvent"
}
},
"name": "person.name",
"surname": "person.surname",
"mail": "person.mail",
"dateOfBirth": {
"": {
"": "person.dateOfBirth"
},
"*": {
"$": "person.dateOfBirth"
}
},
"country": "person.address.country",
"city": "person.address.city"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"dateOfEvent": "=toLong(@(1,dateOfEvent))",
"costsArray": "=split('[,]',@(1,costs))",
"costsJoin": "=join('.',@(1,costsArray))",
"costs": "=toDouble(@(1,costsJoin))",
"incomeArray": "=split('[,]',@(1,income))",
"incomeJoin": "=join('.',@(1,incoemArray))",
"income": "=toDouble(@(1,incomeJoin))",
"eventData": {
"personalData": "=toBoolean(@(1,personalData))"
},
"person": {
"dateOfBirth": "=toLong(@(1,dateOfBirth))"
}
}
},
{
"operation": "default",
"spec": {
"income": null,
"costs": null,
"dateOfEvent": null,
"person": {
"dateOfBirth": null
},
"eventData": {
"eventName": "${eventName}",
"eventId": "${eventId}",
"personalData": "${personalData}"
}
}
},
{
"operation": "remove",
"spec": {
"costsArray": "",
"costsJoin": "",
"incomeArray": "",
"incomeJoin": ""
}
}
]

Everything is fine when I have data like dateOfBirth or income or costs. But when this data are null my json changes its forms and this data ends up at the and and look like this: 

{
"system" : "TGG",
"costs" : 700.0,
"id" : "1",
"person" : {
"name" : "JAMES",
"surname" : "SMITH",
"mail" : "jsmith@gmail.com",
"address" : {
"country" : "US",
"city" : "ORLANDO"
},
"dateOfBirth" : null
},
"eventData" : {
"eventName" : "${eventName}",
"eventId" : "${eventId}",
"personalData" : "${personalData}"
},
"income" : null,
"dateOfEvent" : null
}

I need the json to always look the same, fields to be in the same order, regardless of whether they have values or are null-How can I improve this in my jolt spec?

[#2]

Hi @MWM ,

The reason the order changes when the values are null is because they are added with the default spec after they were removed from the initial shift spec when the values are set to blank. There is no quick fix if you want to maintain the order. I can suggest couple of options:

1- Provide additional shift spec to the end of the above spec to re enforce the desired order. The problem with this is that you have to list all the fields as given in the new structure which tends to be challenging specially when you have a lot of fields and complex structure.

2- Handle the setting of blank values  to null for the desired fields before applying the transformation. For that you can use groovy script to write custom code or take advantage of the UpdateRecord Processor and the powerful nifi record path engine that has a lot of built in functions which can help you do such thing easily. Not only both approaches should be easier than the first option because you dont have to list all fields (only one expected to be blank) but also this will simplify your jolt transformation downstream because you dont need to worry about blank values any more. Im not groovy expert , so I can show you how to do it via the UpdateRecord processor which would look like the following:

Basically I listed the path for each of the desired fields in the input json  and set the value to the following record path function:

/fieldname[not(isEmpty(/fieldname))]

which says give me the value of the given field with the condition that the value is not Empty. the isEmpty function returns true if the value is null or blank. If the condition is not met the returned value will always be null.

Make sure to set the Replacement Value Strategy to Record Path Value.

If that helps please accept solution.

Thanks