Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can we flatten complex JSON file using NIFI.?

Solved Go to solution

Can we flatten complex JSON file using NIFI.?

Super Collaborator

Hi ,Can we flatten complex JSON files (with lists,arrays etc)with NIFI.?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Can we flatten complex JSON file using NIFI.?

@Saikrishna Tarapareddy, Apache NiFi 1.0 includes the JoltTransformJSON processor, which embeds the Jolt JSON-to-JSON transformation library. JoltTransformJSON can indeed be used to flatten JSON.

You can see an example of using the processor here: https://community.hortonworks.com/articles/44726/json-to-json-simplified-with-apache-nifi-and-jolt.h...

And there's an example of a Jolt spec to flatten JSON here: http://jolt-demo.appspot.com/#bucketToPrefixSoup

HDF 2.0, forthcoming, includes Apache NiFi 1.0, so going forward this will likely be the right way to do it. As an alternative, you could use a scripted processor to do something similar. See: http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

9 REPLIES 9

Re: Can we flatten complex JSON file using NIFI.?

@Saikrishna Tarapareddy, Apache NiFi 1.0 includes the JoltTransformJSON processor, which embeds the Jolt JSON-to-JSON transformation library. JoltTransformJSON can indeed be used to flatten JSON.

You can see an example of using the processor here: https://community.hortonworks.com/articles/44726/json-to-json-simplified-with-apache-nifi-and-jolt.h...

And there's an example of a Jolt spec to flatten JSON here: http://jolt-demo.appspot.com/#bucketToPrefixSoup

HDF 2.0, forthcoming, includes Apache NiFi 1.0, so going forward this will likely be the right way to do it. As an alternative, you could use a scripted processor to do something similar. See: http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

Re: Can we flatten complex JSON file using NIFI.?

Super Collaborator

@jfrazee

when I tried to look at these examples , its not showing any JSON or specs. All I get is empty screen like this..

when I tried this link below

http://jolt-demo.appspot.com/#bucketToPrefixSoup

7251-jolt.png

Re: Can we flatten complex JSON file using NIFI.?

Not really sure why it's not loading the examples for you but the JSON input that page should have loaded is:

{
  "Rating": 1,
  "SecondaryRatings": {
    "Design": 4,
    "Price": 2,
    "RatingDimension3": 1
  }
}

And the Jolt spec is:

[
  {
    "operation": "shift",
    "spec": {
      "Rating": "rating-primary",
      //
      // Turn all the SecondaryRatings into prefixed data
      // like "rating-Design" : 4
      "SecondaryRatings": {
        // the "&" in "rating-&" means go up the tree 0 levels,
        // grab what is ther and subtitute it in
        "*": "rating-&"
      }
    }
  }
]

Re: Can we flatten complex JSON file using NIFI.?

If you click on any of the examples,it should fill in the Input and Spec boxes.

Re: Can we flatten complex JSON file using NIFI.?

Super Collaborator

@Matt Burgess

i tried to do that , but it is not filling in the Input or Specs.

Re: Can we flatten complex JSON file using NIFI.?

Here's the example for "prefix soup" (a kind of flattening):

Input:

{
"Rating": 1,
"SecondaryRatings": {
"Design": 4,
"Price": 2,
"RatingDimension3": 1
}
}

Spec:

[
  {
    "operation": "shift",
    "spec": {
      "Rating": "rating-primary",
      //
      // Turn all the SecondaryRatings into prefixed data
      // like "rating-Design" : 4
      "SecondaryRatings": {
        // the "&" in "rating-&" means go up the tree 0 levels,
        // grab what is ther and subtitute it in
        "*": "rating-&"
      }
    }
  }
]

Re: Can we flatten complex JSON file using NIFI.?

Super Collaborator

Hi @Matt Burgess @jfrazee

Where can i find help for JOLT JSON transformations..i am trying to flatten this example json with multi levels.

{

"glossary": {

"title": "example glossary",

"GlossDiv": {

"title": "S",

"GlossList": {

"GlossEntry": {

"ID": "SGML",

"SortAs": "SGML",

"GlossTerm": "Standard Generalized Markup Language",

"Acronym": "SGML",

"Abbrev": "ISO 8879:1986",

"GlossDef": {

"para": "A meta-markup language, used to create markup languages such as DocBook.", "GlossSeeAlso": ["GML", "XML"]

},

"GlossSee": "markup"

}

}

}

}

}

Re: Can we flatten complex JSON file using NIFI.?

Super Collaborator

@Matt Burgess , @jfrazee

I wrote this Shift transformation in NiFi JOLT , which is flattening above JSON..just wanted to make sure of this would be the correct approach..?? Only thing is if we come across a new Tag in input then we will have to write transformations to it.

{ "glossary": { "title": "Glossary Title", "GlossDiv": { "title*": "GlossDivTitles", "GlossList": { "GlossEntry": { "ID": "GlossEntryID", "SortAs": "GlossEntrySortAs", "GlossTerm": "GlossEntryGT", "Acronym": "GlossEntryAcronym", "Abbrev": "GlossEntryAbbr", "GlossDef": { "*" : "GlossDef&" }, "GlossSee": "GlossEntryGSee" } } } } }

Highlighted

Re: Can we flatten complex JSON file using NIFI.?

Don't have an account?
Coming from Hortonworks? Activate your account here