Reply
Explorer
Posts: 29
Registered: ‎01-20-2017

incremental loading technique for Data warehouse

Hi,

    I am trying to undrstand the incrmental loading by modified date.

    I looked at the solution

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_dataintegration/content/incrementally-up...

 

But the solution is like every time you create a full file and then load to warehouse table.

Suppose I have 1 billion record in Reporting table, and suppose there are 1Million record chnages , as per the this incrmental loading logic, we have to create whole 1 billion record file again and create a new table. Also there might be deletes , again create whole file and reload to table. It is line Full load every time. So what happens when the user already looking at the report, what will happen during the record is getting loaded.

 

Is this a perfect design ?

Or are thre any better approaches to do Incremnetal Loading .

Explorer
Posts: 29
Registered: ‎01-20-2017

Re: incremental loading technique for Data warehouse

Hi,

   I could not find any answer for this? Are there any other design  available for OLAP reports in Bigdata while 

   changes has been done in OLTP System.

 

 

Champion
Posts: 710
Registered: ‎05-16-2016

Re: incremental loading technique for Data warehouse

Could you please let me know if you are doing a Hive import or Hdfs using Sqoop

 

Explorer
Posts: 29
Registered: ‎01-20-2017

Re: incremental loading technique for Data warehouse

Hi,

   I am using sqoop and imprting to Hdfs. Then I am making some structural change to data and then loading to Hive.

 

Thanks

 

Announcements