Support Questions

Find answers, ask questions, and share your expertise

Data loss doing incremental load based on a timestamp from SQL Server using QueryDatabaseTableRecord

Explorer

hello all,

 

We are using Nifi to do an incremental load from SQL Server using QueryDatabaseTableRecord where max-value column is updated_at column with a Datetime2 data type. The source tables get updated in milliseconds. We noticed that we are losing few events doing with each QueryDatabaseTableRecord run. As a work-around we created a custom processor that subtracts few seconds from the max-value in the state fetching overlapping events and then we handle any duplicates later on in the pipeline and it seems to solve the issue. However, this is not optimal. We'd like to know what's causing the skips. Our assumption so far is that timestamp datatype accuracy is causing the skips in the incremental load however we're not sure.  

 

1 ACCEPTED SOLUTION

Explorer
1 REPLY 1

Explorer
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.