Should be doable in Spark using the CSV and Avro reader/writer. Your header is quite odd with quoting characters surrounding its column names, so it cannot be understood directly ('"' is an illegal character for an avro field name). We could have the Spark CSV reader ignore this line as a comment since no other line should start with a '"' character. Your data is expressed as quoted values with the quoted character being '|'. Something like the below can achieve a conversion, for CDH5: ~> spark-shell --packages com.databricks:spark-csv_2.10:1.5.0,com.databricks:spark-avro_2.10:4.0.0
> import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType}
> // Manual schema declaration of the 'co' and 'id' column names and types
> val customSchema = StructType(Array(
StructField("co", StringType, true),
StructField("id", IntegerType, true)))
> val df = sqlContext.read.format("com.databricks.spark.csv").option("comment", "\"").option("quote", "|").schema(customSchema).load("/tmp/file.txt")
> df.write.format("com.databricks.spark.avro").save("/tmp/avroout")
> // Note: /tmp/file.txt is input file/dir, and /tmp/avroout is the output dir
... View more
