Member since
11-06-2017
2
Posts
0
Kudos Received
0
Solutions
11-08-2017
03:45 PM
Thanks kgautam for the reply. You said Avro serialized data has no schema saved in the file. This is what I read "Schema is stored along with the Avro data in a file." So basically, if I am using a Avro writer that uses old schema. This same old writer should be able to handle records coming in with new schema (as long as we follow the rules) until I update the writer to use the new schema. If later I update the writer to use the new schema will it be able to give me all the data with new schema? Again all the data with new and old schema are in the same file. I want to understand if I will loose any new fields that I add while writing the data with old schema. Thanks
... View more
11-07-2017
02:27 PM
I have one avro file with first schema then I updated the schema that appends to the same file. So now I have two schemas in one file. How does avro handle this scenario. Will I have any new fields add in the file or will I loose any data while reading this data. This is a real time streaming application where I am writing the data to hdfs. My upstream system might update the schema but the hdfs writer might be on old schema. So the hdfs avro file will have two schemas until I update the writer to handle the newer schema. Note - I don't have schema registry and I am creating one avro file per day. So if a schema is updated in the middle of the day, I will have one avro file with two schemas.
... View more
Labels:
- Labels:
-
Schema Registry