Support Questions
Find answers, ask questions, and share your expertise

Can we flatten complex JSON file using NIFI.?

Solved Go to solution
Highlighted

Can we flatten complex JSON file using NIFI.?

Super Collaborator

Hi ,Can we flatten complex JSON files (with lists,arrays etc)with NIFI.?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Can we flatten complex JSON file using NIFI.?

@Saikrishna Tarapareddy, Apache NiFi 1.0 includes the JoltTransformJSON processor, which embeds the Jolt JSON-to-JSON transformation library. JoltTransformJSON can indeed be used to flatten JSON.

You can see an example of using the processor here: https://community.hortonworks.com/articles/44726/json-to-json-simplified-with-apache-nifi-and-jolt.h...

And there's an example of a Jolt spec to flatten JSON here: http://jolt-demo.appspot.com/#bucketToPrefixSoup

HDF 2.0, forthcoming, includes Apache NiFi 1.0, so going forward this will likely be the right way to do it. As an alternative, you could use a scripted processor to do something similar. See: http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

View solution in original post

9 REPLIES 9
Highlighted

Re: Can we flatten complex JSON file using NIFI.?

@Saikrishna Tarapareddy, Apache NiFi 1.0 includes the JoltTransformJSON processor, which embeds the Jolt JSON-to-JSON transformation library. JoltTransformJSON can indeed be used to flatten JSON.

You can see an example of using the processor here: https://community.hortonworks.com/articles/44726/json-to-json-simplified-with-apache-nifi-and-jolt.h...

And there's an example of a Jolt spec to flatten JSON here: http://jolt-demo.appspot.com/#bucketToPrefixSoup

HDF 2.0, forthcoming, includes Apache NiFi 1.0, so going forward this will likely be the right way to do it. As an alternative, you could use a scripted processor to do something similar. See: http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

View solution in original post

Highlighted

Re: Can we flatten complex JSON file using NIFI.?

Super Collaborator

@jfrazee

when I tried to look at these examples , its not showing any JSON or specs. All I get is empty screen like this..

when I tried this link below

http://jolt-demo.appspot.com/#bucketToPrefixSoup

7251-jolt.png

Highlighted

Re: Can we flatten complex JSON file using NIFI.?

Not really sure why it's not loading the examples for you but the JSON input that page should have loaded is:

{
  "Rating": 1,
  "SecondaryRatings": {
    "Design": 4,
    "Price": 2,
    "RatingDimension3": 1
  }
}

And the Jolt spec is:

[
  {
    "operation": "shift",
    "spec": {
      "Rating": "rating-primary",
      //
      // Turn all the SecondaryRatings into prefixed data
      // like "rating-Design" : 4
      "SecondaryRatings": {
        // the "&" in "rating-&" means go up the tree 0 levels,
        // grab what is ther and subtitute it in
        "*": "rating-&"
      }
    }
  }
]
Highlighted

Re: Can we flatten complex JSON file using NIFI.?

Super Guru

If you click on any of the examples,it should fill in the Input and Spec boxes.

Highlighted

Re: Can we flatten complex JSON file using NIFI.?

Super Collaborator

@Matt Burgess

i tried to do that , but it is not filling in the Input or Specs.

Highlighted

Re: Can we flatten complex JSON file using NIFI.?

Super Guru

Here's the example for "prefix soup" (a kind of flattening):

Input:

{
"Rating": 1,
"SecondaryRatings": {
"Design": 4,
"Price": 2,
"RatingDimension3": 1
}
}

Spec:

[
  {
    "operation": "shift",
    "spec": {
      "Rating": "rating-primary",
      //
      // Turn all the SecondaryRatings into prefixed data
      // like "rating-Design" : 4
      "SecondaryRatings": {
        // the "&" in "rating-&" means go up the tree 0 levels,
        // grab what is ther and subtitute it in
        "*": "rating-&"
      }
    }
  }
]
Highlighted

Re: Can we flatten complex JSON file using NIFI.?

Super Collaborator

Hi @Matt Burgess @jfrazee

Where can i find help for JOLT JSON transformations..i am trying to flatten this example json with multi levels.

{

"glossary": {

"title": "example glossary",

"GlossDiv": {

"title": "S",

"GlossList": {

"GlossEntry": {

"ID": "SGML",

"SortAs": "SGML",

"GlossTerm": "Standard Generalized Markup Language",

"Acronym": "SGML",

"Abbrev": "ISO 8879:1986",

"GlossDef": {

"para": "A meta-markup language, used to create markup languages such as DocBook.", "GlossSeeAlso": ["GML", "XML"]

},

"GlossSee": "markup"

}

}

}

}

}

Highlighted

Re: Can we flatten complex JSON file using NIFI.?

Super Collaborator

@Matt Burgess , @jfrazee

I wrote this Shift transformation in NiFi JOLT , which is flattening above JSON..just wanted to make sure of this would be the correct approach..?? Only thing is if we come across a new Tag in input then we will have to write transformations to it.

{ "glossary": { "title": "Glossary Title", "GlossDiv": { "title*": "GlossDivTitles", "GlossList": { "GlossEntry": { "ID": "GlossEntryID", "SortAs": "GlossEntrySortAs", "GlossTerm": "GlossEntryGT", "Acronym": "GlossEntryAcronym", "Abbrev": "GlossEntryAbbr", "GlossDef": { "*" : "GlossDef&" }, "GlossSee": "GlossEntryGSee" } } } } }

Highlighted

Re: Can we flatten complex JSON file using NIFI.?