Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

CDF/Nifi error with Date field and PutIceberg processor

avatar
Contributor

I have a flow that queries data from a source table and loads it into an iceberg table. The source data includes a Date type which gets read in as a string. The destination table has the same field where the schema is also a Date. When I try to use the PutIceberg processor, I get the following error:

org.apache.nifi.serialization.record.util.IllegalTypeConversionException: 
Failed Conversion of Field [null] from String [2023-10-30] to LocalDate - Caused by: java.lang.NumberFormatException

The flow file is Avro and I'm using an AvroReader service for the PutIceberg. Here's a sample of the Avro:

[ {
"scan_id" : "1698665963-82DD686DBC1979E6FF6C5443BA82456621DB5004",
"queue_seq_id" : 558745163,
"flight_airline_code" : "AC",
"flight_number" : 8663,
"flight_departure_date" : "2023-10-30",
"scan_airport_code" : "YHZ",
"scan_checkpoint_name" : "Transborder",
"scan_stage_name" : "S2 - Screening Lines",
"manual_entry" : "N",
"wait_time_pax_queue" : "Main Queue",
"wait_time_seconds" : 93,
"included_in_counts" : "Y",
"insert_time" : "2023-10-30 13:44:39.0",
"scan_time" : "2023-10-30 11:39:23.0",
"scan_hour" : 11,
"scan_year" : 2023,
"scan_month" : 10,
"scan_day" : 30
}

The issue seems to be the 'flight_departure_date' field. Strangely enough, I have another flow that does essentially the exact same thing and I've had no issues with it there.

1 ACCEPTED SOLUTION

avatar
Contributor

Resolved. The issue was that I was using an ExecuteSQL processor before the PutIceberg processor and I neglected to use logical types. The fix was to switch the property 'Use Avro Logical Types' to true.

View solution in original post

1 REPLY 1

avatar
Contributor

Resolved. The issue was that I was using an ExecuteSQL processor before the PutIceberg processor and I neglected to use logical types. The fix was to switch the property 'Use Avro Logical Types' to true.