question Using JOLT to remove field with null value in Support Questions

Using JOLT to remove field with null value

Adhitya — Mon, 05 Aug 2024 07:00:07 GMT

Hi,

I have a json data as follow.

INPUT :

{
"vlineid" : "9PIA",
"type" : "M",
"catg" : "C",
"nhhigh" : 100.0,
"nhigh" : 50.0,
"nlow" : null,
"nllow" : null,
"uom" : "sec",
"vsectionid" : "PEPO60",
"vparamdesc" : "CYCLE TIME",
"vpartid" : "-",
"vflgipc" : "IPC02",
"vwctid" : "P9PIA0",
"vmachineid" : "IMM04",
"vparam" : "CYCLE-TIME",
"nvalue" : 0.0,
"t" : 1722833150160,
"vcrea" : "NIFI",
"vmodi" : "NIFI",
"dmodi" : "2024-08-05 11:49:17",
"dcrea" : "2024-08-05 11:49:17",
"t_epoch" : 1
}

I tried to remove the null field with this JOLT Specification

JOLT :

[
{
    "operation": "shift",
    "spec": {
      "*": "temp.&"
    }
},
{
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": "=recursivelySquashNulls"
    }
},
{
    "operation": "shift",
    "spec": {
      "temp": ""
    }
},
{
    "operation": "remove",
    "spec": {
      "temp": ""
    }
}
]

OUTPUT :

{
"vlineid" : "9PIA",
"type" : "M",
"catg" : "C",
"nhhigh" : 100.0,
"nhigh" : 50.0,
"uom" : "sec",
"vsectionid" : "PEPO60",
"vparamdesc" : "CYCLE TIME",
"vpartid" : "-",
"vwctid" : "P9PIA0",
"vmachineid" : "IMM04",
"vparam" : "CYCLE-TIME",
"nvalue" : 0.0,
"t" : 1722833150160,
"vcrea" : "NIFI",
"vmodi" : "NIFI",
"dmodi" : "2024-08-05 11:49:17",
"dcrea" : "2024-08-05 11:49:17",
"t_epoch" : 1
}

The JOLT did remove the nlow and nllow field, but also removed the vflgipc when the vflgipc value is not null, and sometimes even the nlow or nllow field was not removed, even though it has a null value. I'm new with JOLT and Record Processing, can someone help me?

Thank You,
Regards,
Muhammad Adhitya Dharamawan

Re: Using JOLT to remove field with null value

SAMSAL — Wed, 07 Aug 2024 09:43:51 GMT

Hi,

You can do this in different ways :

1- Jolt: Squash Null

[ { "operation": "shift", "spec": { "*": "temp.&" } } , { "operation": "modify-overwrite-beta", "spec": { "temp": "=squashNulls" } } , { "operation": "shift", "spec": { "temp": { "*": "&" } } } ]

2- JOLT: using * on the value which will ignore nulls by default.

[ { "operation": "shift", "spec": { "*": { "*": { "$": "&2" } } } } ]

3- JSLT: I like to introduce people to JSTL which is another transformation language supported by Nifi if you are using version 1.16 and above using JSLTTransformJson. Jstl can simplify transformation in some scenarios over jolt. For example in this case the transformation will be as simple as this:

{for(.) .key:.value}

The default filtering applied will take care of this:

If you find this helpful please accept solution.

Thanks

Re: Using JOLT to remove field with null value

VidyaSargur — Mon, 19 Aug 2024 04:52:07 GMT

@Adhitya, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.