Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to automate sqoop incremental import using sqoop job?

Solved Go to solution

How to automate sqoop incremental import using sqoop job?

Explorer

Hi there, How to automate sqoop incremental import using sqoop job? As for as I know, sqoop job remembers the last value. If we create a sqoop job like sqoop job --create myjob -- import --connect blah blah.. sqoop exec myjob and automate the job, It will create the job each time it executes. So, we will get job already exists error. So, Is it possible to automate sqoop incremental import using sqoop job? Please let me know your knowledge on this. Thanks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to automate sqoop incremental import using sqoop job?

Guru

Execute does not create a sqoop job. sqoop job creates the job and you will get an error if you try creating multiple sqoop job with the same job name.

You need to pass --incremental and --check-col on the job to trigger incremental updates.

2 REPLIES 2
Highlighted

Re: How to automate sqoop incremental import using sqoop job?

Guru

Execute does not create a sqoop job. sqoop job creates the job and you will get an error if you try creating multiple sqoop job with the same job name.

You need to pass --incremental and --check-col on the job to trigger incremental updates.

Re: How to automate sqoop incremental import using sqoop job?

Explorer

Hi Ravi,

Thank you very much for your prompt reply and it helped me a lot. I understand the operation now. Once again thanks