Writing avro files with a user defined schema in spark

nicholas_pettyj — Thu, 11 May 2017 23:12:15 GMT

I have a corpus of structured data stored in HDFS as a set of avro files. I need to do some processing to split this set into multiple sets based on the value of a certain field within the data set. This will involve splitting out the individual records based on the data element, bundling them up as new avro files and storing them into separate directories. I have tested a solution with Spark (v2.1.0) using the databricks spark-avro library (v2_11.3.2.0). It performs well, but when I write the data set into new avro files, it applies a spark-avro generated schema. The data types match, but I miss out an certain schema customizations, such as default values and descriptions.

Has anyone successfully applied a user-defined schema when writing avro files with spark-avro (or another similar Spark library)? I have found surprisingly little while searching.

Re: Writing avro files with a user defined schema in spark

abbtah — Tue, 16 May 2017 06:41:27 GMT

A simple Spark app demonstrating how to read and write data in the Parquet and Avro formats.

I hope this can help.

https://github.com/sryza/simplesparkavroapp

question Re: Writing avro files with a user defined schema in spark in Archives of Support Questions (Read Only)

Writing avro files with a user defined schema in spark

Re: Writing avro files with a user defined schema in spark