Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Writing avro files with a user defined schema in spark

Solved Go to solution

Writing avro files with a user defined schema in spark

New Contributor

I have a corpus of structured data stored in HDFS as a set of avro files. I need to do some processing to split this set into multiple sets based on the value of a certain field within the data set. This will involve splitting out the individual records based on the data element, bundling them up as new avro files and storing them into separate directories. I have tested a solution with Spark (v2.1.0) using the databricks spark-avro library (v2_11.3.2.0). It performs well, but when I write the data set into new avro files, it applies a spark-avro generated schema. The data types match, but I miss out an certain schema customizations, such as default values and descriptions.

Has anyone successfully applied a user-defined schema when writing avro files with spark-avro (or another similar Spark library)? I have found surprisingly little while searching.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Writing avro files with a user defined schema in spark

New Contributor

A simple Spark app demonstrating how to read and write data in the Parquet and Avro formats.

I hope this can help.

https://github.com/sryza/simplesparkavroapp

1 REPLY 1
Highlighted

Re: Writing avro files with a user defined schema in spark

New Contributor

A simple Spark app demonstrating how to read and write data in the Parquet and Avro formats.

I hope this can help.

https://github.com/sryza/simplesparkavroapp

Don't have an account?
Coming from Hortonworks? Activate your account here