Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

can we get both newly added and updated records at the same time in sqoop using --incremental import command?

can we get both newly added and updated records at the same time in sqoop using --incremental import command?

New Contributor

I am working on sqoop.

I want newly added and updated records using single command. Is it possible using --incremental import command?

3 REPLIES 3
Highlighted

Re: can we get both newly added and updated records at the same time in sqoop using --incremental import command?

Expert Contributor

Yes, you would use the lastmodified mode for the sqoop incremental update as explained here:

https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_incremental_imports

Re: can we get both newly added and updated records at the same time in sqoop using --incremental import command?

@priyal patel

"--incremental" will import both, updated and inserted rows. You would need to use the "lastmodified" mode. However, a column in the table must exist that contains the row's last modified time.

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

The query would like something like:

sqoop import --connect jdbc:teradata://{host name}/Database=retail 
--connection-manager org.apache.sqoop.teradata.TeradataConnManager 
--username dbc --password dbc --table SOURCE_TBL --target-dir /user/hive/incremental_table -m 1 
--check-column modified_date --incremental lastmodified --last-value {last_import_date}

Once you've imported the data into HDP you can merge it with the existing data using the strategy below:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_dataintegration/content/incrementally-up...

As always, if you find this post helpful, don't forget to "accept" answer.

Re: can we get both newly added and updated records at the same time in sqoop using --incremental import command?

@priyal patel

Don't forget to "accept" an answer or share your findings to close this post. Thanks

Don't have an account?
Coming from Hortonworks? Activate your account here