- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
want to convert csv to nested json using nifi
- Labels:
-
Apache NiFi
Created 08-21-2018 03:49 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
csv file:
"Foo","12","newyork","North avenue","123213"
"Foo1","12","newyork","North avenue","123213"
"Foo2","12","newyork","North avenue","123213"
Output required:
{
"studentName":"Foo",
"Age":"12",
"address__city":"newyork",
"address":{
"address__address1":"North avenue",
"address__zipcode":"123213"
}
}
I am able to convert csv to simple json but not nested json using convertRecord Processor.
please check my Avro schema:
{
"type" : "record",
"name" : "MyClass",
"namespace" : "com.test.avro",
"fields" : [ {
"name" : "studentName",
"type" : "string"
}, {
"name" : "Age",
"type" : "string"
}, {
"name" : "address__city",
"type" : "string"
}, {
"name" : "address",
"type" : {
"type" : "record",
"name" : "address",
"fields" : [ {
"name" : "address__address1",
"type" : "string"
}, {
"name" : "address__zipcode",
"type" : "string"
} ]
}
} ]
}
Created 08-22-2018 03:37 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ConvertRecord is mostly for changing data formats, not structure. UpdateRecord is more appropriate, but I don't believe this is currently possible, as the "address" field doesn't exist in the input, and we don't currently update the schema in that case. I've filed NIFI-5524 to cover this improvement. You'll also be able to accomplish this with JoltTransformRecord when NIFI-5353 is implemented.
In the meantime you can use ConvertRecord to convert from flat CSV into flat JSON (the writer can inherit the record schema), then use JoltTransformJSON to push the fields into the "address" object, here's a spec that will do that:
[ { "operation": "shift", "spec": { "*": { "address1": "[#2].address.address_address1", "zipcode": "[#2].address.address_zipcode", "*": "[#2].&" } } } ]
Created 08-22-2018 03:37 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ConvertRecord is mostly for changing data formats, not structure. UpdateRecord is more appropriate, but I don't believe this is currently possible, as the "address" field doesn't exist in the input, and we don't currently update the schema in that case. I've filed NIFI-5524 to cover this improvement. You'll also be able to accomplish this with JoltTransformRecord when NIFI-5353 is implemented.
In the meantime you can use ConvertRecord to convert from flat CSV into flat JSON (the writer can inherit the record schema), then use JoltTransformJSON to push the fields into the "address" object, here's a spec that will do that:
[ { "operation": "shift", "spec": { "*": { "address1": "[#2].address.address_address1", "zipcode": "[#2].address.address_zipcode", "*": "[#2].&" } } } ]
Created 09-25-2023 05:43 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm new to nifi I'm working on it can you send me the flow file I am getting confused which processor need to use
Created 09-25-2023 02:42 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Abhiram-4455 As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.
Regards,
Diana Torres,Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
Created 08-23-2018 06:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Matt Burgess
Thanks for your help.
It worked and now I am able to convert csv into nested JSON.
Thanks a lot.
