question Re: jolttransformjson - transform some elements but keep others in Archives of Support Questions (Read Only)

jolttransformjson - transform some elements but keep others

mqureshi — Fri, 02 Sep 2016 11:44:53 GMT

I have a complex json and I am using jolttransformjson processor to flatten out the json. I have looked at following links which are very helpful.

https://docs.google.com/presentation/d/1sAiuiFC4Lzz4-064sg1p8EQt2ev0o442MfEbvrpD1ls/edit#slide=id.g9798b391_00

https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/main/java/com/bazaarvoice/jolt/Shiftr.java

But what I still can't find is an example that shows transforming some attributes and not others. So, I have a json and the only thing I want to do really is keep records which contain certain value. Rest will be filtered out. but everything else in the json remains the way it is. Can someone share a quick example of this?

Re: jolttransformjson - transform some elements but keep others

andrewg — Fri, 02 Sep 2016 18:27:08 GMT

Hi, I'm not yet sure if that can be done in JOLT spec. However, how about approaching it from the flow-based programming standpoint? The steps should be:

Transform with JOLT - do all, no filtering
EvaluateJsonPath into an attribute - extract values that will be used to make a decision
RouteOnAttribute - use an expression against the above set of attributes to implement the filtering logic. E.g. discard or re-route records which don't match the criteria.

Re: jolttransformjson - transform some elements but keep others

mqureshi — Fri, 02 Sep 2016 23:54:18 GMT

Thank you @Andrew Grande

Quick question. If I don't need to do any filtering in jolt, why do I need it? To flatten it?

Re: jolttransformjson - transform some elements but keep others

mqureshi — Tue, 06 Sep 2016 07:42:46 GMT

@Andrew Grande

So, I have done step 1 and 2 as you suggested. I am not sure how to implement step 3. Using jolt json, I have flattened my json by moving following objects from array to now as another attribute of the record.

"event_1_detail":{"type":"comma separated values","Description":"<important value>",<many other attributes>}

"event_2_detail":{"type":"comma separated values","Description":"<important value>",<many other attributes>}

"event_3_detail":{"type":"comma separated values","Description":"<important value>",<many other attributes>}

and so on. it's going to be different for each json record.

I have created flow-file attribute using EvalueJson processor which gets the description value. Now out of the elements shown above, I need to drop everything and retain only the one where

"Description":"<the value I want to retain>"

this means of "event_x_detail" above only the one which satisfies my criteria for description should be retained. the rest of json of course still remains there. just rest of "event_x_detail" are dropped. How do i do this using RouteOnAttribute?

Re: jolttransformjson - transform some elements but keep others

mburgess — Sat, 13 May 2017 09:38:51 GMT

In a JOLT spec, if you don't explicitly provide a transformation for a particular field, it will be excluded. So you can include matching rules for the fields you care about (i.e. those that have a certain value), the rest will be discarded. Check the "Filter data from an Array, based on a leaf level value" example at the JOLT Demo online app.

Re: jolttransformjson - transform some elements but keep others

aniruddh02 — Wed, 09 Jan 2019 04:29:29 GMT

If there anyway I can get both transformed and original json out of JoltTransformation processor?

Re: jolttransformjson - transform some elements but keep others

mburgess — Thu, 10 Jan 2019 02:55:18 GMT

Looks like @Bryan Bende answered your question on SO.

Re: jolttransformjson - transform some elements but keep others

UselesssCat — Tue, 31 Mar 2020 15:57:48 GMT

You can use * and & to copy all in a shift operation

Re: jolttransformjson - transform some elements but keep others

UselesssCat — Tue, 31 Mar 2020 15:58:56 GMT

For example I want to transform the zabbix payload from v4.0 to v4.4:

Zabbix Json Payload v4.0 (INPUT)

{ "hosts": [ "Host B", "Zabbix Server" ], "groups": [ "Group X", "Group Y", "Group Z", "Zabbix servers" ], "tags": [ { "tag": "availability", "value": "" }, { "tag": "data center", "value": "Riga" } ], "name": "Either Zabbix agent is unreachable", "clock": 1519304285, "ns": 123456789, "eventid": 42, "value": 1 }

the JOLT transform:

[ { "operation": "shift", "spec": { "hosts": { "*": [ "hosts.[&].host", "hosts.[&].name" ] }, "*": "&" } } ]

The result ( Zabbix v4.4)

{ "hosts" : [ { "host" : "Host B", "name" : "Host B" }, { "host" : "Zabbix Server", "name" : "Zabbix Server" } ], "groups" : [ "Group X", "Group Y", "Group Z", "Zabbix servers" ], "tags" : [ { "tag" : "availability", "value" : "" }, { "tag" : "data center", "value" : "Riga" } ], "name" : "Either Zabbix agent is unreachable", "clock" : 1519304285, "ns" : 123456789, "eventid" : 42, "value" : 1 }