- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Convert from one AVRO schema to another
- Labels:
-
Apache NiFi
Created ‎12-14-2016 08:31 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello there, somehow I dont see AvroRecordConverter in NiFi's available Processors. I need to be able to convert an incoming source file (CSV/XML/JSON) to a generic schema. However, the problem is I am not sure how to - 1. Convert a AVRO 'record' type to another and 2. When using ConvertAvroSchema how do we specify the dynamic properties as specified in the NiFi documentaion here - https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.kite.ConvertAvroSchema/...
Created ‎12-23-2016 05:41 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thank you @Bryan Bende
I managed my use case with Jolt Processor, since AVRO would only transform to the schema of incoming file format, and will adjust only in terms of adding or removing some attributes. However, I was looking for a totally different outbound schema than whats incoming.
Created ‎12-14-2016 02:07 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
To go from CSV or JSON to Avro you would use ConvertJsonToAvro or ConvertCsvToAvro, there is no conversion for XML currently. With ConvertJsonToAvro and ConvertCsvToAvro you can directly enter the schema in the properties of the processor, or you can attempt to use InferAvroSchema before them. InferAvroSchema will guess a schema and put it into a flow file attribute which you can then reference as ${inferred.avro.schema} in the conversion processor.
For ConvertAvroSchema you can either enter schemas directly into the properties, or you can reference flow file attributes using expression language (if you have the schemas in flow file attributes).
Created ‎12-23-2016 05:41 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thank you @Bryan Bende
I managed my use case with Jolt Processor, since AVRO would only transform to the schema of incoming file format, and will adjust only in terms of adding or removing some attributes. However, I was looking for a totally different outbound schema than whats incoming.
