I am trying to undrstand the incrmental loading by modified date.
I looked at the solution
But the solution is like every time you create a full file and then load to warehouse table.
Suppose I have 1 billion record in Reporting table, and suppose there are 1Million record chnages , as per the this incrmental loading logic, we have to create whole 1 billion record file again and create a new table. Also there might be deletes , again create whole file and reload to table. It is line Full load every time. So what happens when the user already looking at the report, what will happen during the record is getting loaded.
Is this a perfect design ?
Or are thre any better approaches to do Incremnetal Loading .
I could not find any answer for this? Are there any other design available for OLAP reports in Bigdata while
changes has been done in OLTP System.
I am using sqoop and imprting to Hdfs. Then I am making some structural change to data and then loading to Hive.