NIFI Convert ISO 8601 Data type and HIVE Streaming

[#1]

Hi Experts

I need some help in handling ISO8601 data type. I get salesforce data in which all the date fields comes in the format as "2015-12-08T08:19:00.000+0000" which is not accepted by Hive3streaming where the target datatype is Timestamp or date. The hive streaming tries to convert to timestamp however throw errors as number format exception.

How can I convert these data type fields automatically?. Each tables at least have 10-20 date fields in this format and i have more than 100 tables. 

I used InvokeHTTP to get the JSON data from Salesforce. Any help on this is really appreciated. 

[#2]

Add a schema and record reader and writer.

You can use UpdateRecord or QueryRecord to tweak formatting if needed

examples

https://www.datainmotion.dev/2020/11/flank-smart-weather-applications-with.html

https://www.datainmotion.dev/2021/01/flank-real-time-transit-information-for.html

https://www.datainmotion.dev/2021/01/flank-using-apache-kudu-as-cache-for.html

https://www.datainmotion.dev/2020/12/smart-stocks-with-flank-nifi-kafka.html

[#3]

@mburgess

The challenge is that i dont want to handle specific columns as i have more than 100 tables and may be 500 Date fields. It has be to generic. The Execute SQL has the logical Data type which converts every datetime to Hive Datetime automatically which works good for SQL Database source. For JSON format and manually setting up each conversion is a tough option.

[#4]

if you add the custom timestamp to the reader and writers it should do this automatically for all timestamp fields