Avro, in general, supports the idea of evolving schemas and I'm trying to support that with an external Hive table. In other words, declare an external hive table and define its schema through the table properties to be read from an HDFS file: TBLPROPERTIES ('avro.schema.url'='hdfs://namenode/common/schemas/schema_v1.avsc') This works fine when all of the files in the directory for the external table are create with schema version 1. However, if i add avro files of version 2 and update the tblproperties accordingly the table becomes unusable. I see errors like this on a select count(*) statement Caused by: org.apache.avro.AvroTypeException: Found com.target.category_data, expecting com.target.category_data
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:231)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:127)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:176)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
at org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader.next(AvroGenericRecordReader.java:153)
at org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader.next(AvroGenericRecordReader.java:52)
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:347) It appears that the Hive Avro serde is using the schema in tblproperties to decode the individual avro files (or more accurately the individual file splits, I supose) instead of the schema in the header of each avro file. I would like for binary avro files created with different avro schemas to be read by the same hive table with a potentially different avro schema. Could anyone suggest ways to do that? I am not opposed to making code changes on the read side (i.e. the hive serde) or the write side (I'm using Storm's avro bolt to write my files). Thanks! -Aaron
... View more
