Was able to make this work with a few modifications... 1) logicalTypes need to be represented as a sub-type like this: {
"name": "session_number",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20
}
} Instead of: {
"name": "session_number",
"type": "bytes",
"logicalType": "decimal",
"precision": 20
} 2) You can't use "?" for missing values because if the missing value is a number (or something other than string) than it will try to parse "?" into the given type and fail. To work around this you can make the fields in the schema nullable by using a union of "null" and the real type. 3) Your input data has the timestamp in a string format, but the schema specifies it as timestamp millis, so the CSV Reader needs the timestamp property set to yyyy-MM-dd'T'HH:mm:ss'Z' Here is the full schema with the changes mentioned in #1 and #2: {
"type": "record",
"name": "page_record",
"fields" : [
{"name": "session_number", "type": { "type": "bytes", "logicalType": "decimal", "precision": 20} },
{"name": "tracking_uuid", "type": "string"},
{"name": "page_location_domain", "type": "string"},
{"name": "page_title", "type": "string"},
{"name": "referring_page_instance_id", "type": ["null", { "type": "bytes", "logicalType": "decimal", "precision": 20} ]},
{"name": "page_instance_id", "type": { "type": "bytes", "logicalType": "decimal", "precision": 20}},
{"name": "event_timestamp", "type": { "type": "long", "logicalType": "timestamp-millis"}},
{"name": "page_location", "type": "string"},
{"name": "attribution_sequence_in_session", "type": "long"},
{"name": "page_sequence_in_session", "type": "long"},
{"name": "page_sequence_in_attribution", "type": "long"},
{"name": "top_level_window_id", "type": "string"},
{"name": "profile_uuid", "type": ["null", "string"]}
]
} And here is the input data with the question marks removed: col_1;col_2;col_3;col_4;col_5;col_6;col_7;col_8;col_9;col_10;col_11;col_12;col_13 36531253;4787ea68-4276-4b3b-b154-d70419d23113;https://www.dummyexample.com;My Dummy website, description;;365311753;2018-01-02T07:08:40Z;https://www.dummyexample.com/fr/openspace/loggin?axes4=priv;1;1;1;_15148769143240.5030172901622478_;
... View more
