Using JOLT to remove field with null value

[#1]

Hi,

I have a json data as follow.

INPUT :

{
  "vlineid" : "9PIA",
  "type" : "M",
  "catg" : "C",
  "nhhigh" : 100.0,
  "nhigh" : 50.0,
  "nlow" : null,
  "nllow" : null,
  "uom" : "sec",
  "vsectionid" : "PEPO60",
  "vparamdesc" : "CYCLE TIME",
  "vpartid" : "-",
  "vflgipc" : "IPC02",
  "vwctid" : "P9PIA0",
  "vmachineid" : "IMM04",
  "vparam" : "CYCLE-TIME",
  "nvalue" : 0.0,
  "t" : 1722833150160,
  "vcrea" : "NIFI",
  "vmodi" : "NIFI",
  "dmodi" : "2024-08-05 11:49:17",
  "dcrea" : "2024-08-05 11:49:17",
  "t_epoch" : 1
}

I tried to remove the null field with this JOLT Specification

JOLT :

[
{
    "operation": "shift",
    "spec": {
      "*": "temp.&"
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": "=recursivelySquashNulls"
    }
  },
  {
    "operation": "shift",
    "spec": {
      "temp": ""
    }
  },
  {
    "operation": "remove",
    "spec": {
      "temp": ""
    }
  }
]

OUTPUT :

{
  "vlineid" : "9PIA",
  "type" : "M",
  "catg" : "C",
  "nhhigh" : 100.0,
  "nhigh" : 50.0,
  "uom" : "sec",
  "vsectionid" : "PEPO60",
  "vparamdesc" : "CYCLE TIME",
  "vpartid" : "-",
  "vwctid" : "P9PIA0",
  "vmachineid" : "IMM04",
  "vparam" : "CYCLE-TIME",
  "nvalue" : 0.0,
  "t" : 1722833150160,
  "vcrea" : "NIFI",
  "vmodi" : "NIFI",
  "dmodi" : "2024-08-05 11:49:17",
  "dcrea" : "2024-08-05 11:49:17",
  "t_epoch" : 1
}

The JOLT did remove the nlow and nllow field, but also removed the vflgipc when the vflgipc value is not null, and sometimes even the nlow or nllow field was not removed, even though it has a null value. I'm new with JOLT and Record Processing, can someone help me?

Thank You,
Regards,
Muhammad Adhitya Dharamawan

[#2 · Accepted solution]

Hi,

You can do this in  different ways :

1- Jolt: Squash Null

[
  {
    "operation": "shift",
    "spec": {
      "*": "temp.&"
    }
  }
  ,
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "temp": "=squashNulls"
    }
  }
  ,
  {
    "operation": "shift",
    "spec": {
      "temp": {
        "*": "&"
      }
    }
  }
]

 2- JOLT: using * on the value which will ignore nulls by default.

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "$": "&2"
        }
      }
    }
  }
]

 3- JSLT: I like to introduce people to JSTL which is another transformation language supported by Nifi if you are using version 1.16 and above using JSLTTransformJson. Jstl can simplify transformation in some scenarios over jolt. For example in this case the transformation will be as simple as this:

{for(.) .key:.value}

The default filtering applied will take care of this:

If you find this helpful please accept solution.

Thanks

[#3]

@Adhitya, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.