03-11-2017 12:59 PM
I am trying to undrstand the incrmental loading by modified date.
I looked at the solution
But the solution is like every time you create a full file and then load to warehouse table.
Suppose I have 1 billion record in Reporting table, and suppose there are 1Million record chnages , as per the this incrmental loading logic, we have to create whole 1 billion record file again and create a new table. Also there might be deletes , again create whole file and reload to table. It is line Full load every time. So what happens when the user already looking at the report, what will happen during the record is getting loaded.
Is this a perfect design ?
Or are thre any better approaches to do Incremnetal Loading .