Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Data will be updated automatically using sqoop?

Data will be updated automatically using sqoop?

New Contributor

Once we import data from RDBMS to HADOOP and after that if data will be updated in RDBMS then data in Hadoop will updated automatically or we have to update it manually??

2 REPLIES 2
Highlighted

Re: Data will be updated automatically using sqoop?

Rising Star

Hi,

Sqoop will add (using --incremental) the new data added to the source RDBMS, but will not update the data already ingested.

You'll need manual process, or third-party tooling to keep your Hadoop copy in sync with the RDBMS.

Re: Data will be updated automatically using sqoop?

Adding to @Christophe Vico's answer above. You can use the below "manual" merge/update strategies after you import the data.

https://www.phdata.io/4-strategies-for-updating-hive-tables/

https://hortonworks.com/blog/four-step-strategy-incremental-updates-hive/

As always, if you find this post helpful, don't forget to "accept" answer.