Created 08-09-2018 11:46 AM
How to do incremental update when I am loading data from SQL server to hive tables using sqoop without creating extra temporary tables ?
Incremental insert is working using below command.
--incremental append --check-column id --last-value 5
But update is not working using below
--incremental lastmodified --check-column UPDATE_DATE --last-value '2018-07-19 16:14:38'
Created 08-09-2018 05:44 PM
Do you see any errors/exceptions? Use double quotes for the date in last-value.
--incremental lastmodified --check-column UPDATE_DATE --last-value "2018-07-19 16:14:38"
Created 08-10-2018 07:15 AM
@Sandeep Nemuri This is also not working. Every time a new record is inserted instead of updating the existing record.
It means the table in hive now has 2 records , both old and new with the same ID
Created 08-13-2018 06:06 PM
@Deb, I think this is expected. Incremental imports mode can be used to retrieve only rows newer than some previously-imported set of rows.
There is no direct way to achieve the use case you are looking for. Having said that you can refer this document : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_data-access/content/incrementally-updati...
Hope this helps.