Skip to main content
Adhitya
Adhitya
Silver ⭐️⭐️
August 5, 2024
Solved

Using JOLT to remove field with null value

Hi,

I have a json data as follow.

INPUT :

{
  "vlineid" : "9PIA",
  "type" : "M",
  "catg" : "C",
  "nhhigh" : 100.0,
  "nhigh" : 50.0,
  "nlow" : null,
  "nllow" : null,
  "uom" : "sec",
  "vsectionid" : "PEPO60",
  "vparamdesc" : "CYCLE TIME",
  "vpartid" : "-",
  "vflgipc" : "IPC02",
  "vwctid" : "P9PIA0",
  "vmachineid" : "IMM04",
  "vparam" : "CYCLE-TIME",
  "nvalue" : 0.0,
  "t" : 1722833150160,
  "vcrea" : "NIFI",
  "vmodi" : "NIFI",
  "dmodi" : "2024-08-05 11:49:17",
  "dcrea" : "2024-08-05 11:49:17",
  "t_epoch" : 1
}

I tried to remove the null field with this JOLT Specification

JOLT :

[
{
    "operation": "shift",
    "spec": {
      "*": "temp.&"
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": "=recursivelySquashNulls"
    }
  },
  {
    "operation": "shift",
    "spec": {
      "temp": ""
    }
  },
  {
    "operation": "remove",
    "spec": {
      "temp": ""
    }
  }
]

OUTPUT :

{
  "vlineid" : "9PIA",
  "type" : "M",
  "catg" : "C",
  "nhhigh" : 100.0,
  "nhigh" : 50.0,
  "uom" : "sec",
  "vsectionid" : "PEPO60",
  "vparamdesc" : "CYCLE TIME",
  "vpartid" : "-",
  "vwctid" : "P9PIA0",
  "vmachineid" : "IMM04",
  "vparam" : "CYCLE-TIME",
  "nvalue" : 0.0,
  "t" : 1722833150160,
  "vcrea" : "NIFI",
  "vmodi" : "NIFI",
  "dmodi" : "2024-08-05 11:49:17",
  "dcrea" : "2024-08-05 11:49:17",
  "t_epoch" : 1
}

The JOLT did remove the nlow and nllow field, but also removed the vflgipc when the vflgipc value is not null, and sometimes even the nlow or nllow field was not removed, even though it has a null value. I'm new with JOLT and Record Processing, can someone help me?

Thank You,
Regards,
Muhammad Adhitya Dharamawan

SAMSAL
SAMSALAnswer
Platinum ⭐️⭐️⭐️
August 7, 2024

Hi,

You can do this in  different ways :

1- Jolt: Squash Null

 

 

[
  {
    "operation": "shift",
    "spec": {
      "*": "temp.&"
    }
  }
  ,
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "temp": "=squashNulls"
    }
  }
  ,
  {
    "operation": "shift",
    "spec": {
      "temp": {
        "*": "&"
      }
    }
  }
]

 

 

 2- JOLT: using * on the value which will ignore nulls by default.

 

 

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "$": "&2"
        }
      }
    }
  }
]

 

 

 3- JSLT: I like to introduce people to JSTL which is another transformation language supported by Nifi if you are using version 1.16 and above using JSLTTransformJson. Jstl can simplify transformation in some scenarios over jolt. For example in this case the transformation will be as simple as this:

 

{for(.) .key:.value}

 

The default filtering applied will take care of this:

SAMSAL_0-1723023722901.png

 

If you find this helpful please accept solution.

Thanks

 

View in thread

2 replies

SAMSAL
SAMSALAnswer
Platinum ⭐️⭐️⭐️
August 7, 2024

Hi,

You can do this in  different ways :

1- Jolt: Squash Null

 

 

[
  {
    "operation": "shift",
    "spec": {
      "*": "temp.&"
    }
  }
  ,
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "temp": "=squashNulls"
    }
  }
  ,
  {
    "operation": "shift",
    "spec": {
      "temp": {
        "*": "&"
      }
    }
  }
]

 

 

 2- JOLT: using * on the value which will ignore nulls by default.

 

 

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "$": "&2"
        }
      }
    }
  }
]

 

 

 3- JSLT: I like to introduce people to JSTL which is another transformation language supported by Nifi if you are using version 1.16 and above using JSLTTransformJson. Jstl can simplify transformation in some scenarios over jolt. For example in this case the transformation will be as simple as this:

 

{for(.) .key:.value}

 

The default filtering applied will take care of this:

SAMSAL_0-1723023722901.png

 

If you find this helpful please accept solution.

Thanks

 

Translate
    VidyaSargur
    Community Manager
    VidyaSargur
    Community Manager
    August 19, 2024

    @Adhitya, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.

    Translate
    Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
    Terms & ConditionsAccessibility statement