Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to automate sqoop incremental import using sqoop job?

avatar
Rising Star

Hi there, How to automate sqoop incremental import using sqoop job? As for as I know, sqoop job remembers the last value. If we create a sqoop job like sqoop job --create myjob -- import --connect blah blah.. sqoop exec myjob and automate the job, It will create the job each time it executes. So, we will get job already exists error. So, Is it possible to automate sqoop incremental import using sqoop job? Please let me know your knowledge on this. Thanks in advance.

1 ACCEPTED SOLUTION

avatar
Guru

Execute does not create a sqoop job. sqoop job creates the job and you will get an error if you try creating multiple sqoop job with the same job name.

You need to pass --incremental and --check-col on the job to trigger incremental updates.

View solution in original post

2 REPLIES 2

avatar
Guru

Execute does not create a sqoop job. sqoop job creates the job and you will get an error if you try creating multiple sqoop job with the same job name.

You need to pass --incremental and --check-col on the job to trigger incremental updates.

avatar
Rising Star

Hi Ravi,

Thank you very much for your prompt reply and it helped me a lot. I understand the operation now. Once again thanks