Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to adapt the schema generated by inverAvroSchema

Solved Go to solution
Highlighted

How to adapt the schema generated by inverAvroSchema

Contributor

Hello,

I use the processor inferavroschema to define the schema of my table SQL then I use PutDatabaseRecord except for some columns I have null values not detectable by inferavroschema For example for a column in my file csv it can have null values but so the schema avro they have string values. During the insertion I have errors I would like to know if there is a way to change the schema Avro without going through a manual writing of the structure of the table with Schema Registry.

Thank you

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to adapt the schema generated by inverAvroSchema

Contributor

I used UpdateAttribut in which I get inferred.avro.schema with getAttribut and apply a 'replace' expression

3 REPLIES 3

Re: How to adapt the schema generated by inverAvroSchema

Hi @Amira khalifa,

Why don't you want to define the schema or your target tables in the schema registry?

You can use InferAvroSchema to help you initializing the schema, then you can adapt it and copy/paste it into the schema registry of your choice. Using InferAvroSchema for every single flow file you'll ingest can be costly in terms of performance and it's only inferring the schema on a subset of data: as you can see, the schema might not be completely correct to take into account all the possibilities.

What you could do, even though it's not what I'd recommend as a long term solution, is to use expression language to update the avro schema generated as an attribute of your flow files.

Hope this helps.

Re: How to adapt the schema generated by inverAvroSchema

Contributor

Hi @Pierre Villard

Thank you for your response.

I want to automate the work as much as possible. For a user of the application I want him to be able to specify only the file to import and then nifi does the necessary. I did a function that created a sql table automatically from inferAvroSchema and I want to adapt this schema. Avro Registry works manually. For each file I have to fill it manually.

Re: How to adapt the schema generated by inverAvroSchema

Contributor

I used UpdateAttribut in which I get inferred.avro.schema with getAttribut and apply a 'replace' expression