jolttransformjson - transform some elements but keep others

[#1]

I have a complex json and I am using jolttransformjson processor to flatten out the json. I have looked at following links which are very helpful.

https://docs.google.com/presentation/d/1sAiuiFC4Lzz4-064sg1p8EQt2ev0o442MfEbvrpD1ls/edit#slide=id.g9...

https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/main/java/com/bazaarvoice/jolt/Shiftr....

But what I still can't find is an example that shows transforming some attributes and not others. So, I have a json and the only thing I want to do really is keep records which contain certain value. Rest will be filtered out. but everything else in the json remains the way it is. Can someone share a quick example of this?

[#2 · Accepted solution]

Hi, I'm not yet sure if that can be done in JOLT spec. However, how about approaching it from the flow-based programming standpoint? The steps should be:

• Transform with JOLT - do all, no filtering
• EvaluateJsonPath into an attribute - extract values that will be used to make a decision
• RouteOnAttribute - use an expression against the above set of attributes to implement the filtering logic. E.g. discard or re-route records which don't match the criteria.

[#3]

Thank you @Andrew Grande

Quick question. If I don't need to do any filtering in jolt, why do I need it? To flatten it?

[#4]

@Andrew Grande

So, I have done step 1 and 2 as you suggested. I am not sure how to implement step 3. Using jolt json, I have flattened my json by moving following objects from array to now as another attribute of the record.

"event_1_detail":{"type":"comma separated values","Description":"<important value>",<many other attributes>}

"event_2_detail":{"type":"comma separated values","Description":"<important value>",<many other attributes>}

"event_3_detail":{"type":"comma separated values","Description":"<important value>",<many other attributes>}

and so on. it's going to be different for each json record.

I have created flow-file attribute using EvalueJson processor which gets the description value. Now out of the elements shown above, I need to drop everything and retain only the one where

"Description":"<the value I want to retain>"

this means of "event_x_detail" above only the one which satisfies my criteria for description should be retained. the rest of json of course still remains there. just rest of "event_x_detail" are dropped. How do i do this using RouteOnAttribute?

[#5]

In a JOLT spec, if you don't explicitly provide a transformation for a particular field, it will be excluded. So you can include matching rules for the fields you care about (i.e. those that have a certain value), the rest will be discarded. Check the "Filter data from an Array, based on a leaf level value" example at the JOLT Demo online app.

[#6]

If there anyway I can get both transformed and original json out of JoltTransformation processor?

,

If there anyway I can get both transformed and original json out of JoltTransformation processor?

[#7]

Looks like @Bryan Bende answered your question on SO.

[#8]

You can use * and & to copy all in a shift operation

[#9]

For example I want to transform the zabbix payload from v4.0 to v4.4:

Zabbix Json Payload v4.0 (INPUT)

{
  "hosts": [
    "Host B",
    "Zabbix Server"
  ],
  "groups": [
    "Group X",
    "Group Y",
    "Group Z",
    "Zabbix servers"
  ],
  "tags": [
    {
      "tag": "availability",
      "value": ""
    },
    {
      "tag": "data center",
      "value": "Riga"
    }
  ],
  "name": "Either Zabbix agent is unreachable",
  "clock": 1519304285,
  "ns": 123456789,
  "eventid": 42,
  "value": 1
}

the JOLT transform:

[
  {
    "operation": "shift",
    "spec": {
      "hosts": {
        "*": [
          "hosts.[&].host",
          "hosts.[&].name"
        ]
      },
      "*": "&"
    }
  }
]

 The result ( Zabbix v4.4)

{
  "hosts" : [ {
    "host" : "Host B",
    "name" : "Host B"
  }, {
    "host" : "Zabbix Server",
    "name" : "Zabbix Server"
  } ],
  "groups" : [ "Group X", "Group Y", "Group Z", "Zabbix servers" ],
  "tags" : [ {
    "tag" : "availability",
    "value" : ""
  }, {
    "tag" : "data center",
    "value" : "Riga"
  } ],
  "name" : "Either Zabbix agent is unreachable",
  "clock" : 1519304285,
  "ns" : 123456789,
  "eventid" : 42,
  "value" : 1
}