Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

jolttransformjson - transform some elements but keep others

Solved Go to solution
Highlighted

jolttransformjson - transform some elements but keep others

Super Guru

I have a complex json and I am using jolttransformjson processor to flatten out the json. I have looked at following links which are very helpful.

https://docs.google.com/presentation/d/1sAiuiFC4Lzz4-064sg1p8EQt2ev0o442MfEbvrpD1ls/edit#slide=id.g9...

https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/main/java/com/bazaarvoice/jolt/Shiftr....

But what I still can't find is an example that shows transforming some attributes and not others. So, I have a json and the only thing I want to do really is keep records which contain certain value. Rest will be filtered out. but everything else in the json remains the way it is. Can someone share a quick example of this?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: jolttransformjson - transform some elements but keep others

Hi, I'm not yet sure if that can be done in JOLT spec. However, how about approaching it from the flow-based programming standpoint? The steps should be:

  • Transform with JOLT - do all, no filtering
  • EvaluateJsonPath into an attribute - extract values that will be used to make a decision
  • RouteOnAttribute - use an expression against the above set of attributes to implement the filtering logic. E.g. discard or re-route records which don't match the criteria.

View solution in original post

8 REPLIES 8
Highlighted

Re: jolttransformjson - transform some elements but keep others

Hi, I'm not yet sure if that can be done in JOLT spec. However, how about approaching it from the flow-based programming standpoint? The steps should be:

  • Transform with JOLT - do all, no filtering
  • EvaluateJsonPath into an attribute - extract values that will be used to make a decision
  • RouteOnAttribute - use an expression against the above set of attributes to implement the filtering logic. E.g. discard or re-route records which don't match the criteria.

View solution in original post

Highlighted

Re: jolttransformjson - transform some elements but keep others

Super Guru

Thank you @Andrew Grande

Quick question. If I don't need to do any filtering in jolt, why do I need it? To flatten it?

Highlighted

Re: jolttransformjson - transform some elements but keep others

Super Guru

@Andrew Grande

So, I have done step 1 and 2 as you suggested. I am not sure how to implement step 3. Using jolt json, I have flattened my json by moving following objects from array to now as another attribute of the record.

"event_1_detail":{"type":"comma separated values","Description":"<important value>",<many other attributes>}

"event_2_detail":{"type":"comma separated values","Description":"<important value>",<many other attributes>}

"event_3_detail":{"type":"comma separated values","Description":"<important value>",<many other attributes>}

and so on. it's going to be different for each json record.

I have created flow-file attribute using EvalueJson processor which gets the description value. Now out of the elements shown above, I need to drop everything and retain only the one where

"Description":"<the value I want to retain>"

this means of "event_x_detail" above only the one which satisfies my criteria for description should be retained. the rest of json of course still remains there. just rest of "event_x_detail" are dropped. How do i do this using RouteOnAttribute?

Highlighted

Re: jolttransformjson - transform some elements but keep others

Super Guru

In a JOLT spec, if you don't explicitly provide a transformation for a particular field, it will be excluded. So you can include matching rules for the fields you care about (i.e. those that have a certain value), the rest will be discarded. Check the "Filter data from an Array, based on a leaf level value" example at the JOLT Demo online app.

Highlighted

Re: jolttransformjson - transform some elements but keep others

New Contributor

If there anyway I can get both transformed and original json out of JoltTransformation processor?

,

If there anyway I can get both transformed and original json out of JoltTransformation processor?

Highlighted

Re: jolttransformjson - transform some elements but keep others

Super Guru

Looks like @Bryan Bende answered your question on SO.

Highlighted

Re: jolttransformjson - transform some elements but keep others

New Contributor

You can use * and & to copy all in a shift operation

Highlighted

Re: jolttransformjson - transform some elements but keep others

New Contributor

For example I want to transform the zabbix payload from v4.0 to v4.4:

 

Zabbix Json Payload v4.0 (INPUT)

 

{
  "hosts": [
    "Host B",
    "Zabbix Server"
  ],
  "groups": [
    "Group X",
    "Group Y",
    "Group Z",
    "Zabbix servers"
  ],
  "tags": [
    {
      "tag": "availability",
      "value": ""
    },
    {
      "tag": "data center",
      "value": "Riga"
    }
  ],
  "name": "Either Zabbix agent is unreachable",
  "clock": 1519304285,
  "ns": 123456789,
  "eventid": 42,
  "value": 1
}

 

 

the JOLT transform:

 

[
  {
    "operation": "shift",
    "spec": {
      "hosts": {
        "*": [
          "hosts.[&].host",
          "hosts.[&].name"
        ]
      },
      "*": "&"
    }
  }
]

 

 The result ( Zabbix v4.4)

 

{
  "hosts" : [ {
    "host" : "Host B",
    "name" : "Host B"
  }, {
    "host" : "Zabbix Server",
    "name" : "Zabbix Server"
  } ],
  "groups" : [ "Group X", "Group Y", "Group Z", "Zabbix servers" ],
  "tags" : [ {
    "tag" : "availability",
    "value" : ""
  }, {
    "tag" : "data center",
    "value" : "Riga"
  } ],
  "name" : "Either Zabbix agent is unreachable",
  "clock" : 1519304285,
  "ns" : 123456789,
  "eventid" : 42,
  "value" : 1
}

 

Don't have an account?
Coming from Hortonworks? Activate your account here