Can we flatten complex JSON file using NIFI.?

[#1]

Hi ,Can we flatten complex JSON files (with lists,arrays etc)with NIFI.?

[#2 · Accepted solution]

@Saikrishna Tarapareddy, Apache NiFi 1.0 includes the JoltTransformJSON processor, which embeds the Jolt JSON-to-JSON transformation library. JoltTransformJSON can indeed be used to flatten JSON.

You can see an example of using the processor here: https://community.hortonworks.com/articles/44726/json-to-json-simplified-with-apache-nifi-and-jolt.h...

And there's an example of a Jolt spec to flatten JSON here: http://jolt-demo.appspot.com/#bucketToPrefixSoup

HDF 2.0, forthcoming, includes Apache NiFi 1.0, so going forward this will likely be the right way to do it. As an alternative, you could use a scripted processor to do something similar. See: http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

[#3]

@jfrazee

when I tried to look at these examples , its not showing any JSON or specs. All I get is empty screen like this..

when I tried this link below

http://jolt-demo.appspot.com/#bucketToPrefixSoup

[#4]

Not really sure why it's not loading the examples for you but the JSON input that page should have loaded is:

{
  "Rating": 1,
  "SecondaryRatings": {
    "Design": 4,
    "Price": 2,
    "RatingDimension3": 1
  }
}

And the Jolt spec is:

[
  {
    "operation": "shift",
    "spec": {
      "Rating": "rating-primary",
      //
      // Turn all the SecondaryRatings into prefixed data
      // like "rating-Design" : 4
      "SecondaryRatings": {
        // the "&" in "rating-&" means go up the tree 0 levels,
        // grab what is ther and subtitute it in
        "*": "rating-&"
      }
    }
  }
]

[#5]

If you click on any of the examples,it should fill in the Input and Spec boxes.

[#6]

@Matt Burgess

i tried to do that , but it is not filling in the Input or Specs.

[#7]

Here's the example for "prefix soup" (a kind of flattening):

Input:

{
"Rating": 1,
"SecondaryRatings": {
"Design": 4,
"Price": 2,
"RatingDimension3": 1
}
}

Spec:

[
  {
    "operation": "shift",
    "spec": {
      "Rating": "rating-primary",
      //
      // Turn all the SecondaryRatings into prefixed data
      // like "rating-Design" : 4
      "SecondaryRatings": {
        // the "&" in "rating-&" means go up the tree 0 levels,
        // grab what is ther and subtitute it in
        "*": "rating-&"
      }
    }
  }
]

[#8]

Hi @Matt Burgess @jfrazee

Where can i find help for JOLT JSON transformations..i am trying to flatten this example json with multi levels.

{

"glossary": {

"title": "example glossary",

"GlossDiv": {

"title": "S",

"GlossList": {

"GlossEntry": {

"ID": "SGML",

"SortAs": "SGML",

"GlossTerm": "Standard Generalized Markup Language",

"Acronym": "SGML",

"Abbrev": "ISO 8879:1986",

"GlossDef": {

"para": "A meta-markup language, used to create markup languages such as DocBook.", "GlossSeeAlso": ["GML", "XML"]

},

"GlossSee": "markup"

}

}

}

}

}

[#9]

@Matt Burgess , @jfrazee

I wrote this Shift transformation in NiFi JOLT , which is flattening above JSON..just wanted to make sure of this would be the correct approach..?? Only thing is if we come across a new Tag in input then we will have to write transformations to it.

{ "glossary": { "title": "Glossary Title", "GlossDiv": { "title*": "GlossDivTitles", "GlossList": { "GlossEntry": { "ID": "GlossEntryID", "SortAs": "GlossEntrySortAs", "GlossTerm": "GlossEntryGT", "Acronym": "GlossEntryAcronym", "Abbrev": "GlossEntryAbbr", "GlossDef": { "*" : "GlossDef&" }, "GlossSee": "GlossEntryGSee" } } } } }

[#10]

+ @jfrazee @Matt Burgess