Created 10-30-2023 07:42 AM
I have a flow that queries data from a source table and loads it into an iceberg table. The source data includes a Date type which gets read in as a string. The destination table has the same field where the schema is also a Date. When I try to use the PutIceberg processor, I get the following error:
org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
Failed Conversion of Field [null] from String [2023-10-30] to LocalDate - Caused by: java.lang.NumberFormatException
The flow file is Avro and I'm using an AvroReader service for the PutIceberg. Here's a sample of the Avro:
[ {
"scan_id" : "1698665963-82DD686DBC1979E6FF6C5443BA82456621DB5004",
"queue_seq_id" : 558745163,
"flight_airline_code" : "AC",
"flight_number" : 8663,
"flight_departure_date" : "2023-10-30",
"scan_airport_code" : "YHZ",
"scan_checkpoint_name" : "Transborder",
"scan_stage_name" : "S2 - Screening Lines",
"manual_entry" : "N",
"wait_time_pax_queue" : "Main Queue",
"wait_time_seconds" : 93,
"included_in_counts" : "Y",
"insert_time" : "2023-10-30 13:44:39.0",
"scan_time" : "2023-10-30 11:39:23.0",
"scan_hour" : 11,
"scan_year" : 2023,
"scan_month" : 10,
"scan_day" : 30
}
The issue seems to be the 'flight_departure_date' field. Strangely enough, I have another flow that does essentially the exact same thing and I've had no issues with it there.
Created 10-30-2023 07:48 AM
Resolved. The issue was that I was using an ExecuteSQL processor before the PutIceberg processor and I neglected to use logical types. The fix was to switch the property 'Use Avro Logical Types' to true.
Created 10-30-2023 07:48 AM
Resolved. The issue was that I was using an ExecuteSQL processor before the PutIceberg processor and I neglected to use logical types. The fix was to switch the property 'Use Avro Logical Types' to true.