Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Is it possible to do an incremental import using Sqoop free form query?

avatar
New Contributor

Please point me to documentation for the same. More helpful if I get examples as well

1 ACCEPTED SOLUTION

avatar
New Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
7 REPLIES 7

avatar

@Suresh Babu Shanmugam

An Incremental Import in Sqoop is easily replicated in free form query

For example let’s assume you have the following table and the last import was on “2013-01-25” (column: createdOn) which corresponds to the “id” 3 (column:id):

2925-screen-shot-2016-03-21-at-52308-pm.png

You can import the changes/increment since that time using any of the following two “Incremental Update Import” commands (the first by id and the second by last modification date):

2926-screen-shot-2016-03-21-at-52800-pm.png

The equivalent “Free Form Query Import” commands would be:

2927-screen-shot-2016-03-21-at-52958-pm.png

So, if you require flexibility with your incremental import then expand/modify on the free form query above. Just keep in mind that the flag ’--m 1’ signifies that only 1 mapper will be used and jobs will run in serial form. This can be changed as needed.

For information on Incremental Imports see:

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

For information on Free Form Query see:

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_free_form_query_imports

For advanced examples of Free Form Query Imports I strongly recommend looking at:

https://www.safaribooksonline.com/library/view/apache-sqoop-cookbook/9781449364618/ch04.html

avatar
New Contributor

Hi Eyad Garelnabi,

I apologize for the delay. Your solution works. Thank you.

Actually we are trying Join queries. Till now we are not successful with incremental import.

I will definitely post the solution once we crack join queries incremental import.

Thanks once again,

Suresh

avatar

Great! Looking forward to seeing your final solution 🙂

avatar

@Suresh Babu Shanmugam I find using free form query to be more effective, because you can control exact interval/range for which you want to bring data.

avatar
New Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
New Contributor

@Suresh Babu Shanmugam I had the same problem and your solution worked for me. Thank you.

avatar
Master Collaborator

the free query form for lastmodified is not correct, because you can manage the updates rows, and also, you will get duplicate rows, so i dont recomend this solution alternative

109882-captura.jpg


2927-screen-shot-2016-03-21-at-52958-pm.png