Support Questions
Find answers, ask questions, and share your expertise

want to convert csv to nested json using nifi

Solved Go to solution
Highlighted

want to convert csv to nested json using nifi

csv file:

"Foo","12","newyork","North avenue","123213"
"Foo1","12","newyork","North avenue","123213"
"Foo2","12","newyork","North avenue","123213"
Output required:
{
    "studentName":"Foo",
    "Age":"12",
    "address__city":"newyork",
    "address":{
        "address__address1":"North avenue",
        "address__zipcode":"123213"
    }
}
I am able to convert csv to simple json but not nested json using convertRecord Processor.
please check my Avro schema:


{
  "type" : "record",
  "name" : "MyClass",
  "namespace" : "com.test.avro",
  "fields" : [ {
    "name" : "studentName",
    "type" : "string"
  }, {
    "name" : "Age",
    "type" : "string"
  }, {
    "name" : "address__city",
    "type" : "string"
  }, {
    "name" : "address",
    "type" : {
      "type" : "record",
      "name" : "address",
      "fields" : [ {
        "name" : "address__address1",
        "type" : "string"
      }, {
        "name" : "address__zipcode",
        "type" : "string"
      } ]
    }
  } ]
}	
	

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: want to convert csv to nested json using nifi

Super Guru

ConvertRecord is mostly for changing data formats, not structure. UpdateRecord is more appropriate, but I don't believe this is currently possible, as the "address" field doesn't exist in the input, and we don't currently update the schema in that case. I've filed NIFI-5524 to cover this improvement. You'll also be able to accomplish this with JoltTransformRecord when NIFI-5353 is implemented.

In the meantime you can use ConvertRecord to convert from flat CSV into flat JSON (the writer can inherit the record schema), then use JoltTransformJSON to push the fields into the "address" object, here's a spec that will do that:

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "address1": "[#2].address.address_address1",
        "zipcode": "[#2].address.address_zipcode",
        "*": "[#2].&"
      }
    }
  }
]

View solution in original post

2 REPLIES 2
Highlighted

Re: want to convert csv to nested json using nifi

Super Guru

ConvertRecord is mostly for changing data formats, not structure. UpdateRecord is more appropriate, but I don't believe this is currently possible, as the "address" field doesn't exist in the input, and we don't currently update the schema in that case. I've filed NIFI-5524 to cover this improvement. You'll also be able to accomplish this with JoltTransformRecord when NIFI-5353 is implemented.

In the meantime you can use ConvertRecord to convert from flat CSV into flat JSON (the writer can inherit the record schema), then use JoltTransformJSON to push the fields into the "address" object, here's a spec that will do that:

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "address1": "[#2].address.address_address1",
        "zipcode": "[#2].address.address_zipcode",
        "*": "[#2].&"
      }
    }
  }
]

View solution in original post

Highlighted

Re: want to convert csv to nested json using nifi

Hi Matt Burgess

Thanks for your help.

It worked and now I am able to convert csv into nested JSON.

Thanks a lot.

Don't have an account?