I am looking for some suggestions and knowledge on streaming job failures and restarts.   I created a Nifi flow that captures changes in Source DB and push them to destination DB (a DWH) the changes were captured using capturechangeMysql processor.. which runs 24/7 , so i call it similar to a streaming service.. During development whenever the job fails for what ever reasons, I stop the job, clear its state and fixes issues and reruns job again. But when it goes live, whenever there is a failure of job, between the time i fix and rerun the job, all the changes that were happeneing to source were missed or lost.. imagine if the complex issue took a day to fix, all the data for 24 hours would be lost at destination.   I would like to know about different options on how to handle these situations and nullify data loss ? I tried the option of "Retrieve All Records" in the CaptureChangeMySQL 1.18.0 processor. but it is filling up entire queues and taking forver to process latest records.   Can some please throw some light here..   ... View more

Hi Team,   I am desperately looking for a helping hand here. My flow will infuse/replicate data from different small databases into a single big database. I am using changecapturemysql processor to capture changes in small databases and putdatabase record to put into destination big database. The table structure is same for all databases. but in order to differentiate in the big database there will be an additional column.  Example: (let me take only a simple and single table in all databases) in sources, city_name is the PK. in destination,  city_name and db_name  both will form the PK.   DB1.city table city_name New york Mumbai   DB2.city table city_name Paris New york     Destination DB city table city_name    db_name New york.     DB1 Mumbai        DB1 Paris             DB2 New york.     DB2   I am able to capture the Inserts and Updates and put them in destination database. but when I delete a record in source, nothing is happening in the destination database. There is no error also. How can I achieve that when i delete a record in any source DB, it needs to delete that particular record in destination DB. How to make nifi identify the record properly to delete. I also wonder how Nifi able to identify the record that need to be updated but not the record that need to be deleted. Please help. ... View more