I am looking for some suggestions and knowledge on streaming job failures and restarts.
I created a Nifi flow that captures changes in Source DB and push them to destination DB (a DWH)
the changes were captured using capturechangeMysql processor.. which runs 24/7 , so i call it similar to a streaming service.. During development whenever the job fails for what ever reasons, I stop the job, clear its state and fixes issues and reruns job again.
But when it goes live, whenever there is a failure of job, between the time i fix and rerun the job, all the changes that were happeneing to source were missed or lost.. imagine if the complex issue took a day to fix, all the data for 24 hours would be lost at destination.
I would like to know about different options on how to handle these situations and nullify data loss ?
I tried the option of "Retrieve All Records" in the CaptureChangeMySQL 1.18.0 processor. but it is filling up entire queues and taking forver to process latest records.
Can some please throw some light here..
