Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Can we flatten complex JSON file using NIFI.?

Super Collaborator

Hi ,Can we flatten complex JSON files (with lists,arrays etc)with NIFI.?

1 ACCEPTED SOLUTION

@Saikrishna Tarapareddy, Apache NiFi 1.0 includes the JoltTransformJSON processor, which embeds the Jolt JSON-to-JSON transformation library. JoltTransformJSON can indeed be used to flatten JSON.

You can see an example of using the processor here: https://community.hortonworks.com/articles/44726/json-to-json-simplified-with-apache-nifi-and-jolt.h...

And there's an example of a Jolt spec to flatten JSON here: http://jolt-demo.appspot.com/#bucketToPrefixSoup

HDF 2.0, forthcoming, includes Apache NiFi 1.0, so going forward this will likely be the right way to do it. As an alternative, you could use a scripted processor to do something similar. See: http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

View solution in original post

9 REPLIES 9

@Saikrishna Tarapareddy, Apache NiFi 1.0 includes the JoltTransformJSON processor, which embeds the Jolt JSON-to-JSON transformation library. JoltTransformJSON can indeed be used to flatten JSON.

You can see an example of using the processor here: https://community.hortonworks.com/articles/44726/json-to-json-simplified-with-apache-nifi-and-jolt.h...

And there's an example of a Jolt spec to flatten JSON here: http://jolt-demo.appspot.com/#bucketToPrefixSoup

HDF 2.0, forthcoming, includes Apache NiFi 1.0, so going forward this will likely be the right way to do it. As an alternative, you could use a scripted processor to do something similar. See: http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

Super Collaborator

@jfrazee

when I tried to look at these examples , its not showing any JSON or specs. All I get is empty screen like this..

when I tried this link below

http://jolt-demo.appspot.com/#bucketToPrefixSoup

7251-jolt.png

Not really sure why it's not loading the examples for you but the JSON input that page should have loaded is:

{
  "Rating": 1,
  "SecondaryRatings": {
    "Design": 4,
    "Price": 2,
    "RatingDimension3": 1
  }
}

And the Jolt spec is:

[
  {
    "operation": "shift",
    "spec": {
      "Rating": "rating-primary",
      //
      // Turn all the SecondaryRatings into prefixed data
      // like "rating-Design" : 4
      "SecondaryRatings": {
        // the "&" in "rating-&" means go up the tree 0 levels,
        // grab what is ther and subtitute it in
        "*": "rating-&"
      }
    }
  }
]

Super Guru

If you click on any of the examples,it should fill in the Input and Spec boxes.

Super Collaborator

@Matt Burgess

i tried to do that , but it is not filling in the Input or Specs.

Super Guru

Here's the example for "prefix soup" (a kind of flattening):

Input:

{
"Rating": 1,
"SecondaryRatings": {
"Design": 4,
"Price": 2,
"RatingDimension3": 1
}
}

Spec:

[
  {
    "operation": "shift",
    "spec": {
      "Rating": "rating-primary",
      //
      // Turn all the SecondaryRatings into prefixed data
      // like "rating-Design" : 4
      "SecondaryRatings": {
        // the "&" in "rating-&" means go up the tree 0 levels,
        // grab what is ther and subtitute it in
        "*": "rating-&"
      }
    }
  }
]

Super Collaborator

Hi @Matt Burgess @jfrazee

Where can i find help for JOLT JSON transformations..i am trying to flatten this example json with multi levels.

{

"glossary": {

"title": "example glossary",

"GlossDiv": {

"title": "S",

"GlossList": {

"GlossEntry": {

"ID": "SGML",

"SortAs": "SGML",

"GlossTerm": "Standard Generalized Markup Language",

"Acronym": "SGML",

"Abbrev": "ISO 8879:1986",

"GlossDef": {

"para": "A meta-markup language, used to create markup languages such as DocBook.", "GlossSeeAlso": ["GML", "XML"]

},

"GlossSee": "markup"

}

}

}

}

}

Super Collaborator

@Matt Burgess , @jfrazee

I wrote this Shift transformation in NiFi JOLT , which is flattening above JSON..just wanted to make sure of this would be the correct approach..?? Only thing is if we come across a new Tag in input then we will have to write transformations to it.

{ "glossary": { "title": "Glossary Title", "GlossDiv": { "title*": "GlossDivTitles", "GlossList": { "GlossEntry": { "ID": "GlossEntryID", "SortAs": "GlossEntrySortAs", "GlossTerm": "GlossEntryGT", "Acronym": "GlossEntryAcronym", "Abbrev": "GlossEntryAbbr", "GlossDef": { "*" : "GlossDef&" }, "GlossSee": "GlossEntryGSee" } } } } }