Support Questions
Find answers, ask questions, and share your expertise

can we get both newly added and updated records at the same time in sqoop using --incremental import command?

Explorer

I am working on sqoop.

I want newly added and updated records using single command. Is it possible using --incremental import command?

5 REPLIES 5

Expert Contributor

Yes, you would use the lastmodified mode for the sqoop incremental update as explained here:

https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_incremental_imports

@priyal patel

"--incremental" will import both, updated and inserted rows. You would need to use the "lastmodified" mode. However, a column in the table must exist that contains the row's last modified time.

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

The query would like something like:

sqoop import --connect jdbc:teradata://{host name}/Database=retail 
--connection-manager org.apache.sqoop.teradata.TeradataConnManager 
--username dbc --password dbc --table SOURCE_TBL --target-dir /user/hive/incremental_table -m 1 
--check-column modified_date --incremental lastmodified --last-value {last_import_date}

Once you've imported the data into HDP you can merge it with the existing data using the strategy below:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_dataintegration/content/incrementally-up...

As always, if you find this post helpful, don't forget to "accept" answer.

New Contributor

Hello,

I have a question: What if we did not have a column in the table  that contains the row's last modified time, or generally we did not know which column would be changed.

New Contributor

Hello,

I have a question: What if we did not have a column in the table  that contains the row's last modified time, or generally we did not know which column would be changed.

@priyal patel

Don't forget to "accept" an answer or share your findings to close this post. Thanks

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.