I am looking for some suggestions and knowledge on streaming job failures and restarts.   I created a Nifi flow that captures changes in Source DB and push them to destination DB (a DWH) the changes were captured using capturechangeMysql processor.. which runs 24/7 , so i call it similar to a streaming service.. During development whenever the job fails for what ever reasons, I stop the job, clear its state and fixes issues and reruns job again. But when it goes live, whenever there is a failure of job, between the time i fix and rerun the job, all the changes that were happeneing to source were missed or lost.. imagine if the complex issue took a day to fix, all the data for 24 hours would be lost at destination.   I would like to know about different options on how to handle these situations and nullify data loss ? I tried the option of "Retrieve All Records" in the CaptureChangeMySQL 1.18.0 processor. but it is filling up entire queues and taking forver to process latest records.   Can some please throw some light here..   ... View more

I have a mysql destination table with column type datetime. my nifi flow is like convertJsontoSQL -> putsql (into destination table) The action is delete for putsql. after conversion of jsontosql i am getting a value '2023-03-28 15:54:28' which is a valid timestamp. but the flow is failing with below error:   duetoThevalue of the sqlargs. 14.value is '2023-03-28 15:54:28', which cannot be converted to a timestamp; routing to failure: java.sql.SQLDataException: The value of the sql.args. 14.value is '2023-03-28 15:54:28', which cannot be converted to a timestamp - Caused by: java. text. ParseException: Unparseable date: "2023-03-28 15:54:28"   I can able to use the same "2023-03-28 15:54:28" for manually inserting into a sample datetime column. and can able to delete a record with where clause as this value. I could not understand why Nifi is still not able to convert it to timestamp.   Any suggestions are appreciated. ... View more