Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

incremental loading technique for Data warehouse

incremental loading technique for Data warehouse

Explorer

Hi,

    I am trying to undrstand the incrmental loading by modified date.

    I looked at the solution

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_dataintegration/content/incrementally-up...

 

But the solution is like every time you create a full file and then load to warehouse table.

Suppose I have 1 billion record in Reporting table, and suppose there are 1Million record chnages , as per the this incrmental loading logic, we have to create whole 1 billion record file again and create a new table. Also there might be deletes , again create whole file and reload to table. It is line Full load every time. So what happens when the user already looking at the report, what will happen during the record is getting loaded.

 

Is this a perfect design ?

Or are thre any better approaches to do Incremnetal Loading .

3 REPLIES 3

Re: incremental loading technique for Data warehouse

Explorer

Hi,

   I could not find any answer for this? Are there any other design  available for OLAP reports in Bigdata while 

   changes has been done in OLTP System.

 

 

Re: incremental loading technique for Data warehouse

Champion

Could you please let me know if you are doing a Hive import or Hdfs using Sqoop

 

Highlighted

Re: incremental loading technique for Data warehouse

Explorer

Hi,

   I am using sqoop and imprting to Hdfs. Then I am making some structural change to data and then loading to Hive.

 

Thanks