Member since
08-09-2024
2
Posts
2
Kudos Received
0
Solutions
08-11-2024
02:57 AM
2 Kudos
Thank you @SAMSAL .. appreciated the way of explanation.. generally i'l prefer to add/update the max.initialvalue.column property of QDT to restart fetching missed data as a one time modification but I would love to see NIFI using fault tolerant mechanism in future so that the developers don't have to find the workaround or have to recover the lost data manually by modifying the QDT
... View more
08-10-2024
12:15 AM
Hello there.. I am new to Nifi and have started using QueryDatabaseTable for incremental fetch. I want to know how to handle a situation as below. For example QDT processor fetched new data based on id column - id 50 to id 55 State of QDT processor (max value QDT processor processed/View State) - 55 Next Steps - ConvertAvrotoJson----->ConvertJsontoSQL ---->PutSQL Now what if the PutSQL processor failed for some reason ? Should not the QDT processor roll back the state of max value from 55 to 49 so that next time QDT processor executes it again starts fetching id from 50 ? When I checked the state of QDT processor it showed me 55 although the new records were not inserted in final table and when i execute the QDT processor again it fetched data from 56 causing data loss at the destination side.. Is there any way to let the QDT processor know that the downstream processors failed and it should now roll back the state to previous value before the current run ?
... View more
Labels:
- Labels:
-
Apache NiFi