question Re: Can we flatten complex JSON file using NIFI.? in Archives of Support Questions (Read Only)

Can we flatten complex JSON file using NIFI.?

saikrishna_tara — Thu, 01 Sep 2016 03:53:53 GMT

Hi ,Can we flatten complex JSON files (with lists,arrays etc)with NIFI.?

Re: Can we flatten complex JSON file using NIFI.?

jfrazee — Thu, 01 Sep 2016 04:03:19 GMT

@Saikrishna Tarapareddy, Apache NiFi 1.0 includes the JoltTransformJSON processor, which embeds the Jolt JSON-to-JSON transformation library. JoltTransformJSON can indeed be used to flatten JSON.

You can see an example of using the processor here: https://community.hortonworks.com/articles/44726/json-to-json-simplified-with-apache-nifi-and-jolt.html

And there's an example of a Jolt spec to flatten JSON here: http://jolt-demo.appspot.com/#bucketToPrefixSoup

HDF 2.0, forthcoming, includes Apache NiFi 1.0, so going forward this will likely be the right way to do it. As an alternative, you could use a scripted processor to do something similar. See: http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

Re: Can we flatten complex JSON file using NIFI.?

saikrishna_tara — Mon, 19 Aug 2019 09:36:05 GMT

@jfrazee

when I tried to look at these examples , its not showing any JSON or specs. All I get is empty screen like this..

when I tried this link below

http://jolt-demo.appspot.com/#bucketToPrefixSoup

Re: Can we flatten complex JSON file using NIFI.?

jfrazee — Fri, 02 Sep 2016 01:21:21 GMT

Not really sure why it's not loading the examples for you but the JSON input that page should have loaded is:

{
  "Rating": 1,
  "SecondaryRatings": {
    "Design": 4,
    "Price": 2,
    "RatingDimension3": 1
  }
}

And the Jolt spec is:

[
  {
    "operation": "shift",
    "spec": {
      "Rating": "rating-primary",
      //
      // Turn all the SecondaryRatings into prefixed data
      // like "rating-Design" : 4
      "SecondaryRatings": {
        // the "&" in "rating-&" means go up the tree 0 levels,
        // grab what is ther and subtitute it in
        "*": "rating-&"
      }
    }
  }
]

Re: Can we flatten complex JSON file using NIFI.?

mburgess — Fri, 02 Sep 2016 01:23:25 GMT

If you click on any of the examples,it should fill in the Input and Spec boxes.

Re: Can we flatten complex JSON file using NIFI.?

saikrishna_tara — Fri, 02 Sep 2016 04:40:04 GMT

@Matt Burgess

i tried to do that , but it is not filling in the Input or Specs.

Re: Can we flatten complex JSON file using NIFI.?

mburgess — Fri, 02 Sep 2016 04:50:32 GMT

Here's the example for "prefix soup" (a kind of flattening):

Input:

{
"Rating": 1,
"SecondaryRatings": {
"Design": 4,
"Price": 2,
"RatingDimension3": 1
}
}

Spec:

[
  {
    "operation": "shift",
    "spec": {
      "Rating": "rating-primary",
      //
      // Turn all the SecondaryRatings into prefixed data
      // like "rating-Design" : 4
      "SecondaryRatings": {
        // the "&" in "rating-&" means go up the tree 0 levels,
        // grab what is ther and subtitute it in
        "*": "rating-&"
      }
    }
  }
]

Re: Can we flatten complex JSON file using NIFI.?

saikrishna_tara — Sat, 03 Sep 2016 03:16:22 GMT

Hi @Matt Burgess @jfrazee

Where can i find help for JOLT JSON transformations..i am trying to flatten this example json with multi levels.

{

"glossary": {

"title": "example glossary",

"GlossDiv": {

"title": "S",

"GlossList": {

"GlossEntry": {

"ID": "SGML",

"SortAs": "SGML",

"GlossTerm": "Standard Generalized Markup Language",

"Acronym": "SGML",

"Abbrev": "ISO 8879:1986",

"GlossDef": {

"para": "A meta-markup language, used to create markup languages such as DocBook.", "GlossSeeAlso": ["GML", "XML"]

"GlossSee": "markup"

}

Re: Can we flatten complex JSON file using NIFI.?

saikrishna_tara — Tue, 06 Sep 2016 23:35:09 GMT

@Matt Burgess , @jfrazee

I wrote this Shift transformation in NiFi JOLT , which is flattening above JSON..just wanted to make sure of this would be the correct approach..?? Only thing is if we come across a new Tag in input then we will have to write transformations to it.

{ "glossary": { "title": "Glossary Title", "GlossDiv": { "title*": "GlossDivTitles", "GlossList": { "GlossEntry": { "ID": "GlossEntryID", "SortAs": "GlossEntrySortAs", "GlossTerm": "GlossEntryGT", "Acronym": "GlossEntryAcronym", "Abbrev": "GlossEntryAbbr", "GlossDef": { "*" : "GlossDef&" }, "GlossSee": "GlossEntryGSee" } } } } }

Re: Can we flatten complex JSON file using NIFI.?

SQLShaw — Fri, 09 Sep 2016 20:06:43 GMT

+ @jfrazee @Matt Burgess