Jolt quick reference for Nifi Jolt Processors

wcbdata · ‎06-10-2019

OBJECTIVE:

Provide a quick-start guide for using the Jolt language within a NiFi JoltTransform (JoltTransformJSON or JoltTransformRecord).

OVERVIEW:

The NiFi JoltTransform uses the powerful Jolt language to parse JSON. Combined with the NiFi Schema Registry, this gives NiFi the ability to traverse, recurse, transform, and modify nearly any data format that can be described in AVRO or, using JSON as an intermediary step.

Although the language itself is open-source, and some documentation is available in the JavaDoc, this article can provide a starting point for understanding basic Jolt operations.

PREREQUISITES:

HDF 3.0 or later (NiFi 1.2.0.3 or later)

BASICS OF JOLT:

Simplified Overview
1. The JoltTransform applies a set of transformations described in a JSON specification to an input JSON document and generates a new output JSON document.
Jolt Specification
1. Overview
  A Jolt Specification is a JSON structure that contains two root elements:
  - operation (string): shift, sort, cardinality, modify-default-beta, modify-overwrite-beta, modify-define-beta, or remove
  - spec (JSON): A set of key/value pairs of the form {“input-side search”: “output-side transformation”}.
2. Simple: Select a single jolt transform type from the drop-down, then type or paste the specification JSON
3. Chained: Multiple Jolt specifications can be chained together sequentially in an array of simple specifications
Stock Transforms
- Shift: Read values or portions of the input JSON tree and add them to specified locations in the output.
  - Example: I have a bunch of things in the breadbox that I want to move to the countertop. Let’s move everything in the breadbox to the countertop:

Input:

{
   "breadbox": {
    "loaf1": {
       "type": "white"
    },
    "loaf2": {
       "type": "wheat"
    }
  },
  "fridge": {
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly"
    }
  }
}

Spec:

[
  {
     "operation": "shift",
    "spec": {
       "breadbox": "counterTop"
    }
    }
]

Output:

{
   "counterTop": {
    "loaf1": {
       "type": "white"
    },
    "loaf2": {
       "type": "wheat"
    }
  }
}

Default: Non-destructively adds values or arrays of values to the output JSON.
- Example: I want to slice up loaf1of bread if it exists. Let’s add an array of slices to loaf1:

Input:

 {
     "counterTop": {
       "loaf1": {
         "type": "white"
       },
       "loaf2": {
         "type": "wheat"
       },
       "jar1": {
         "contents": "peanut butter"
       },
       "jar2": {
         "contents": "jelly"
       }
     }

Spec:

 [
     {
       "operation": "default",
       "spec": {
         "counterTop": {
           "loaf1": {
             "slices": [
               "slice1",
               "slice2",
               "slice3",
               "slice4
             ]
           }
         }
       }
     }
   ]
 }

Output:

{
   "counterTop" : {
    "loaf1" : {
      "type" : "white",
       "slices" : [ "slice1", "slice2", "slice3", "slice4" ]
    },
    "loaf2" : {
      "type" : "wheat"
    },
    "jar1" : {
       "contents" : "peanut butter"
    },
    "jar2" : {
       "contents" : "jelly"
    }
  }
}

Cardinality: Transforms elements in the input JSON to single values or to arrays (lists) in the output.
- Example: I have too many slices of bread. No matter how many there are, I just want the first one in the array, but as a single value:

Input:

{
   "counterTop": {
    "loaf1": {
       "type": "white",
       "slices": [
         "slice1",
        "slice2",
         "slice3",
         "slice4"
      ]
    },
    "loaf2": {
       "type": "wheat"
    },
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly"
    }
  }
}

Spec:

[
   {
      "operation": "cardinality",
     "spec": {
        "counterTop": {
          "loaf1": {
            "slices": "ONE"
         }
       }
     }
    }
  ]

Output:

{
   "counterTop" : {
    "loaf1" : {
      "type" : "white",
       "slices" : "slice1"
    },
    "loaf2" : {
      "type" : "wheat"
    },
    "jar1" : {
       "contents" : "peanut butter"
    },
    "jar2" : {
       "contents" : "jelly"
    }
  }
}

Remove: Remove elements if found in the input JSON.
- Example: I don’t really want loaf2 or jar1 (who needs whole wheat bread or peanut butter when you have jelly on pain bread!). Let’s remove loaf2 and jar2:

Input:

{
   "counterTop": {
    "loaf1": {
       "type": "white",
       "slices": "slice1"
    },
    "loaf2": {
       "type": "wheat"
    },
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly"
    }
  }
}

Spec:

[
   {
      "operation": "remove",
     "spec": {
        "counterTop": {
          "loaf2": "",
          "jar1": ""
       }
     }
    }
  ]

Output:

{
   "counterTop" : {
    "loaf1" : {
      "type" : "white",
       "slices" : "slice1"
    },
    "jar2" : {
      "contents" : "jelly"
    }
  }
}

Modify: Write calculated values to elements in the target JSON. Calculations include basic string and math operations (toLower, toUpper, concat, min/max/abs, toInteger, toDouble, toLong and can be applied to source JSON values.
- Example: I really like jelly. Let’s make whatever’s in jar1 ALL CAPS so we can shout about it!

Input:

{
   "counterTop": {
    "loaf1": {
       "type": "white",
       "slices": "slice1"
    },
    "jar2": {
       "contents": "jelly"
    }
  }
}

Spec:

[
  {
     "operation": "modify-overwrite-beta",
    "spec": {
       "counterTop": {
         "jar2": {
           "contents": "=toUpper"
        }
      }
    }
  }
]

Output:

{
  "counterTop" : {
    "loaf1" : {
      "type" : "white",
       "slices" : "slice1"
    },
    "jar2" : {
       "contents" : "JELLY"
    }
  }
}

Sort: Sorts all arrays and maps from the input JSON into the output. Sort cannot be configured beyond this all-or-nothing sort. Let’s put the jelly first to make it easier to spread on the bread later:
- Example: Let's sort the ingredients so that the jelly comes first. Jelly is more important, and it will be easier to spread that way.

Input:

{
   "counterTop": {
    "loaf1": {
       "type": "white",
       "slices": "slice1"
    },
    "jar2": {
       "contents": "JELLY"
    }
  }
}

Spec:

[
  {
     "operation": "sort"
    }
  ]

Output:

{
   "counterTop" : {
    "jar2" : {
       "contents" : "JELLY"
    },
    "loaf1" : {
       "slices" : "slice1",
      "type" : "white"
    }
  }
}

Custom Transforms: (Custom Transforms are out of scope for this tutorial)

Wildcards and Operators
1. Input-side (lefthand side)
  Input-side wildcards retrieve a value or JSON tree from the input JSON.
  - * (asterisk)
    The asterisk wildcard traverses and reads each element in the source JSON at the level of the preceding search specification. Typically, the asterisk wildcard will return an array of elements.
    Example: Rather than just one source element, such as the breadbox, let’s grab everything, no matter what element it’s in, and put it on the counter

Input:

{
   "breadbox": {
    "loaf1": {
       "type": "white"
    },
    "loaf2": {
       "type": "wheat"
    }
  },
  "fridge": {
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly"
    }
  }
}

Spec:

[
  {
     "operation": "shift",
    "spec": {
      "*": "counterTop"
    }
    }
  ]

Output:

{
   "counterTop" : [ {
    "loaf1" : {
      "type" : "white"
    },
    "loaf2" : {
      "type" : "wheat"
    }
  }, {
    "jar1" : {
       "contents" : "peanut butter"
    },
    "jar2" : {
      "contents" : "jelly"
    }
  } ]
}

The asterisk wildcard can be used with other string characters to parse data within an input JSON element (we’ll use the $ wildcard notation here – see below for an explanation of that)
Example: Let’s take a look at the expiration date on the jelly. I am not a stickler for expiration dates, so I just want to check the year:

Input:

{
   "breadbox": {
    "loaf1": {
       "type": "white"
    },
    "loaf2": {
       "type": "wheat"
    }
  },
  "fridge": {
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly",
       "expiration": "25-APR-2019"
    }
  }
}

Spec:

[
  {
     "operation": "shift",
    "spec": {
       "fridge": {
         "jar2": {
           "expiration": {
             "*-*-*": {
               "$(0,3)": "expiry.year"
            }
          }
        }
      }
    }
    }
  ]

Output:

{
  "expiry" : {
    "year" : "2019"
  }
}

@ (“at” or arobase)
The “at” wildcard traverses backwards up the source JSON and returns the entire tree or value at the specified position.
- @ or @0 (return value or tree of the matched key from the input JSON)
  Example: Let’s say I take a look at the jelly in jar2, and it has spoiled – We can use @contents to just toss the jelly into the garbage, but if the jelly is terribly bad, we can use @ or @0 to throw out everything in the jar, @1 to throw out everything in the fridge, or @2 to toss the whole kitchen into the garbage!

Input:

{
   "breadbox": {
    "loaf1": {
       "type": "white"
    },
    "loaf2": {
       "type": "wheat"
    }
  },
  "fridge": {
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly",
       "expiration": "25-APR-2019"
    }
  }
}

Spec:

[
  {
     "operation": "shift",
    "spec": {
       "fridge": {
         "jar2": {
           "contents": "garbage1",
           "@0": "garbage2",
           "@1": "garbage3",
           "@2": "garbage4"
        }
      }
    }
    }
  ]

Output:

{
  "garbage0" : {
     "contents" : "jelly",
     "expiration" : "25-APR-2019"
  },
  "garbage1" : "jelly",
  "garbage2" : {
     "contents" : "jelly",
     "expiration" : "25-APR-2019"
  },
  "garbage3" : {
    "jar1" : {
       "contents" : "peanut butter"
    },
    "jar2" : {
       "contents" : "jelly",
       "expiration" : "25-APR-2019"
    }
  },
  "garbage4" : {
     "breadbox" : {
       "loaf1" : {
         "type" : "white"
      },
       "loaf2" : {
         "type" : "wheat"
      }
    },
    "fridge" : {
      "jar1" : {
         "contents" : "peanut butter"
      },
      "jar2" : {
         "contents" : "jelly",
         "expiration" : "25-APR-2019"
      }
    }
  }
}

Output-side (righthand side)
Output-side wildcards return a single value that can be used in a target JSON key, key path or value.
- & (ampersand)
  1. The ampersand wildcard traverses backwards up the source JSON tree, beginning at the level of the preceding match. It returns only the value or key name (not the tree). The ampersand can be used in three ways:
    1. & or &0 (return the name of the matched key from the input JSON)
      Example: Let’s look at jar2 more closely, but I only care about what’s in it. We’ll just put the value of “contents” into the same element’s name (@0)

Input:

{
   "breadbox": {
    "loaf1": {
       "type": "white"
    },
    "loaf2": {
       "type": "wheat"
    }
  },
  "fridge": {
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly",
       "expiration": "25-APR-2019"
    }
  }
}

Spec:

[
  {
     "operation": "shift",
    "spec": {
       "fridge": {
         "jar2": {
           "contents": "&0"
        }
      }
    }
    }
  ]

Output:

{
  "contents" : "jelly"
}

&n (walk back up the tree ‘n’ levels and return the key name from the specified level)
Example: Since the extra “contents” key is a bit superfluous, let’s just use the name of the parent element (&1) instead:

Input:

{
   "breadbox": {
    "loaf1": {
       "type": "white"
    },
    "loaf2": {
       "type": "wheat"
    }
  },
  "fridge": {
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly",
       "expiration": "25-APR-2019"
    }
  }
}

Spec:

[
  {
     "operation": "shift",
    "spec": {
       "fridge": {
         "jar2": {
           "contents": "&1"
        }
      }
    }
    }
  ]

Output:

{
  "jar2" : "jelly"
}

&(n,x) (walk back up the tree ‘n’ levels and return the key name from the x^th child of the key at that level)
Example: We really just want to know where to look for jelly, no matter what container it’s in. Let’s look at the top-level parent’s name instead (&(2,0))

Input:

{
   "breadbox": {
    "loaf1": {
       "type": "white"
    },
    "loaf2": {
       "type": "wheat"
    }
  },
  "fridge": {
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly",
       "expiration": "25-APR-2019"
    }
  }
}

Spec:

[
  {
     "operation": "shift",
    "spec": {
       "fridge": {
         "jar2": {
           "contents": "&(2,0)"
        }
      }
    }
    }
  ]

Output:

{
  "fridge" : "jelly"
}

@ (“at” or arobase)

The “at” wildcard traverses backwards up the source JSON and returns the entire tree or value at the specified position. Same functionality as on the input side, above: @, @(n), @(keyName), and @(n,keyName) forms
Example: Let’s see what we have in all the jars in our refrigerator. We want to match everything with a name starting with “jar” (jar*) and return the contents of each element we find (@0,contents):

Input:

{
   "breadbox": {
    "loaf1": {
       "type": "white"
    },
    "loaf2": {
       "type": "wheat"
    }
  },
  "fridge": {
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly",
       "expiration": "25-APR-2019"
    }
  }
}

Spec:

[
  {
     "operation": "shift",
    "spec": {
      "*": {
         "jar*": {
           "@(0,contents)": "Things in jars"
        }
      }
    }
    }
  ]

Output:

{
  "Things in jars" : [ "peanut butter", "jelly" ]
}

$ (dollar sign)
- The dollar sign traverses backwards up the source JSON and returns only the value at the specified position.
  Same functionality as @, above: $, $(n), and $(n,x) forms

Input:

{
   "breadbox": {
    "loaf1": {
       "type": "white"
    },
    "loaf2": {
       "type": "wheat"
    }
  },
  "fridge": {
    "jar1": {
       "contents": "peanut butter"
    },
    "jar2": {
       "contents": "jelly",
       "expiration": "25-APR-2019"
    }
  }
}

Spec:

[
  {
     "operation": "shift",
    "spec": {
      "*": {
         "jar*": {
           "$0": "List of jars"
        }
      }
    }
    }
  ]

Output:

{
  "List of jars" : [ "jar1", "jar2" ]
}

“Temp” workspace
In a chained Jolt specification, it is possible to create a temporary structure as a workspace within the output JSON. This temporary structure can be useful for making multi-pass transformations or for holding a copy of the original input JSON during destructive transformations. They can then be removed from the output JSON within the same chained specification before the output JSON is produced.

For an example, see the JOLT transform for this article:
https://community.hortonworks.com/articles/232333/image-data-flow-for-industrial-imaging.html

In this example spec, three “shift” operations are chained together. The “particles-orig” element is created to back up the original data in “particles,” then three passes are attempted because there may be a variable number of semicolon-delimited values in the “particles” value. When the three passes are complete, the successful pass is written to the output as “particles” and the backup is removed with a “remove” operation.

Chained Spec:

[
  {
    "operation": "shift",
    "spec": {
      "particles": ["particles-orig",
                  "particles-0",
                  "particles-1",
                  "particles-2",
                  "particles-3",
                  "particles-4"],
      "timestamp": "ts",
      "*": "&"
    }
                  },
  {
    "operation": "shift",
    "spec": {
      "particles-orig": "particles-orig",
      "particles-0": {
        "*;*;*;*;*": {
          "$(0,1)": "tmp.particle1[]",
          "$(0,2)": "tmp.particle2[]",
          "$(0,3)": "tmp.particle3[]",
          "$(0,4)": "tmp.particle4[]",
          "$(0,5)": "tmp.particle5[]"
        }
      },
      "particles-1": {
        "*;*;*;*": {
          "$(0,1)": "tmp.particle1[]",
          "$(0,2)": "tmp.particle2[]",
          "$(0,3)": "tmp.particle3[]",
          "$(0,4)": "tmp.particle4[]"
        }
      },
      "particles-2": {
        "*;*;*": {
          "$(0,1)": "tmp.particle1[]",
           "$(0,2)": "tmp.particle2[]",
          "$(0,3)": "tmp.particle3[]"
        }
      },
      "particles-3": {
        "*;*": {
          "$(0,1)": "tmp.particle1[]",
          "$(0,2)": "tmp.particle2[]"
        }
      },
      "particles-4": "tmp.particle1[]",
      "*": "&"
    }
                  },
  {
    "operation": "shift",
    "spec": {
      "tmp": {
        "*": {
          "0": {
            "*,*,*,*": {
              "@(4,runid)": "particles.[#4].runid",
              "@(4,ts)": "particles.[#4].ts",
              "$(0,1)": "particles.[#4].Xloc",
              "$(0,2)": "particles.[#4].Yloc",
              "$(0,3)": "particles.[#4].Xdim",
              "$(0,4)": "particles.[#4].Ydim"
            }
          }
        }
      },
      "*": "&"
    }
                  },
  {
    "operation": "remove",
    "spec": {
      "particles-orig": ""
    }
                  }
                  ]

REFERENCES:

RELATED POSTS:

Image Data Flow for Industrial Imaging

peter1_biro · ‎07-01-2019

Hello,

thanks for perfect examples!
What would be the jolt specification for this input/output look, please? The number of tags can be dynamic and delimiter is always colon char.

Input:

{
  "log": {
    "vector": "tag1:tag2:tag3"
  }
}

Spec:

???

Output:

{
  "tags" : [ "tag1","tag2","tag3" ]
}

Thanks

turker_tunali · ‎07-01-2019

I really appreciated your work. I bookmarked this page.

PVVK · ‎06-02-2020

@peter1_biro

[
  {
    "operation": "shift",
    "spec": {
      "log": {
        "vector": "tags"
      }
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "tags": "=split(':',@0)"
    }
  }
]

PVVK · ‎09-27-2020

Hi @wcbdata

Can you explain the usage of '#' in the spec you used above:

{
    "operation": "shift",
    "spec": {
      "tmp": {
        "*": {
          "0": {
            "*,*,*,*": {
              "@(4,runid)": "particles.[#4].runid",
              "@(4,ts)": "particles.[#4].ts",
              "$(0,1)": "particles.[#4].Xloc",
              "$(0,2)": "particles.[#4].Yloc",
              "$(0,3)": "particles.[#4].Xdim",
              "$(0,4)": "particles.[#4].Ydim"
            }
          }
        }
      },
      "*": "&"
    }
 }

Jolt quick reference for Nifi Jolt Processors

Apache NiFi

Simplified Overview

Jolt Specification

Stock Transforms

Wildcards and Operators

Input-side (lefthand side)

Output-side (righthand side)

Re: Jolt quick reference for Nifi Jolt Processors

Re: Jolt quick reference for Nifi Jolt Processors

Re: Jolt quick reference for Nifi Jolt Processors

Re: Jolt quick reference for Nifi Jolt Processors