Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Convert JSON to CSV with Map

Highlighted

Convert JSON to CSV with Map

New Contributor

I read a lot about convertig json to csv or the other way around. But somehow it seems like I'm still doing something wrong. My JSON, which I get, looks like this:

{
  "YLMMR" : [ {
    "stunde" : "2019-01-22 13:00:00",
    "bstid" : "YLMMR",
    "height" : "372.00",
    "temperature" : "-5.80",
    "wind" : "10.080",
    "gusts" : "17.280",
    "snowfall" : "0.00",
    "rain" : "0.00",
    "snowheight" : "0.70"
  },
  {
    "stunde" : "2019-01-22 14:00:00",
    "bstid" : "YLMMR",
    "height" : "386.40",
    "temperature" : "-1.80",
    "wind" : "6.120",
    "gusts" : "29.880",
    "snowfall" : "0.00",
    "rain" : "0.00",
    "snowheight" : "1.00"
  }],
  "YLLCI" : [ {
    "stunde" : "2019-01-22 13:00:00",
    "bstid" : "YLLCI",
    "height" : "386.40",
    "temperature" : "-1.80",
    "wind" : "6.120",
    "gusts" : "29.880",
    "snowfall" : "0.00",
    "rain" : "0.00",
    "snowheight" : "1.00"
  },
  {
    "stunde" : "2019-01-22 14:00:00",
    "bstid" : "YLLCI",
    "height" : "386.40",
    "temperature" : "-1.80",
    "wind" : "6.120",
    "gusts" : "29.880",
    "snowfall" : "0.00",
    "rain" : "0.00",
    "snowheight" : "1.00"
  } ]
}

YLMMR and YLLCI are only two of many values which are all different. So I thought I need my AvroSchema to define a record with map inside and inside the map records. So I defined my schema like this:

{
  "name": "wetter",
  "type": "record",
  "fields": [{
        "name": "bst",
        "type": {
              "type": "map",
              "values": {
                "type": "array",
                "items": {
                  "name": "datum",
                  "type": "record",
                  "fields": [
                    {"name": "stunde", "type": "string"},
                    {"name": "bstid", "type": "string"},
                    {"name": "height", "type": "string"},
                    {"name": "temperature", "type": "string"},
                    {"name": "wind", "type": "string"},
                    {"name": "gusts", "type": "string"},
                    {"name": "snowfall", "type": "string"},
                    {"name": "rain", "type": "string"},
                    {"name": "snowheight", "type": "string"}
                  ]
                }
              }
            }
        }
    ]
}

My Nifi-job looks like the attachment "nifi-job.png"

98382-nifi-job.png

And I have a JsonTreeReader as well as a CSVRecordSetWriter. If any more information are needed, then please let me know. The file that comes out of the convertion processor contains just "bst" and nothing else.

Best regards
Marco

Don't have an account?
Coming from Hortonworks? Activate your account here