Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

incremental update in hive using sqoop

Highlighted

incremental update in hive using sqoop

How to do incremental update when I am loading data from SQL server to hive tables using sqoop without creating extra temporary tables ?

Incremental insert is working using below command.

--incremental append --check-column id --last-value 5

But update is not working using below

--incremental lastmodified --check-column UPDATE_DATE --last-value '2018-07-19 16:14:38'

3 REPLIES 3
Highlighted

Re: incremental update in hive using sqoop

@Deb

Do you see any errors/exceptions? Use double quotes for the date in last-value.

--incremental lastmodified --check-column UPDATE_DATE --last-value "2018-07-19 16:14:38"

Re: incremental update in hive using sqoop

@Sandeep Nemuri This is also not working. Every time a new record is inserted instead of updating the existing record.

It means the table in hive now has 2 records , both old and new with the same ID

Highlighted

Re: incremental update in hive using sqoop

@Deb, I think this is expected. Incremental imports mode can be used to retrieve only rows newer than some previously-imported set of rows.

There is no direct way to achieve the use case you are looking for. Having said that you can refer this document : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_data-access/content/incrementally-updati...

Hope this helps.

Don't have an account?
Coming from Hortonworks? Activate your account here