Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Convert from one AVRO schema to another

avatar
Contributor

Hello there, somehow I dont see AvroRecordConverter in NiFi's available Processors. I need to be able to convert an incoming source file (CSV/XML/JSON) to a generic schema. However, the problem is I am not sure how to - 1. Convert a AVRO 'record' type to another and 2. When using ConvertAvroSchema how do we specify the dynamic properties as specified in the NiFi documentaion here - https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.kite.ConvertAvroSchema/...

1 ACCEPTED SOLUTION

avatar
Contributor

Thank you @Bryan Bende

I managed my use case with Jolt Processor, since AVRO would only transform to the schema of incoming file format, and will adjust only in terms of adding or removing some attributes. However, I was looking for a totally different outbound schema than whats incoming.

View solution in original post

2 REPLIES 2

avatar
Master Guru

To go from CSV or JSON to Avro you would use ConvertJsonToAvro or ConvertCsvToAvro, there is no conversion for XML currently. With ConvertJsonToAvro and ConvertCsvToAvro you can directly enter the schema in the properties of the processor, or you can attempt to use InferAvroSchema before them. InferAvroSchema will guess a schema and put it into a flow file attribute which you can then reference as ${inferred.avro.schema} in the conversion processor.

For ConvertAvroSchema you can either enter schemas directly into the properties, or you can reference flow file attributes using expression language (if you have the schemas in flow file attributes).

avatar
Contributor

Thank you @Bryan Bende

I managed my use case with Jolt Processor, since AVRO would only transform to the schema of incoming file format, and will adjust only in terms of adding or removing some attributes. However, I was looking for a totally different outbound schema than whats incoming.