Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop incremental import issue

Highlighted

Sqoop incremental import issue

Contributor

Hi,

We are using HDP-2.5.3 and facing issues in sqoop incremental import. Detailed explanation as below:

1) Used sqoop to import initial data.

2) In source (Oracle), we have some changes like 1 data is inserted and 2 data got updated.

3) Using incremental we are trying to get the changes merged but its not happening.

Screenshot for updated data:

15626-capture.jpg

Sqoop command for incremental load. But while using this command data is not getting merged. Note that from above screen shot record witth value 8000 is the new record and other 2 are updated records.

sqoop import -Dmapreduce.job.user.classpath.first=true --connect jdbc:oracle:thin:@//3.209.124.204:1521/eqlmpoc  --fetch-size 1000 --username utility_test --password utility_test --table UTILITY_TEST.EMP_DETAILS --check-column HIREDATE --incremental lastmodified --last-value "2017-05-28 16:01:12" --target-dir  hdfs://nn.tcsgegdc.com:8020/data/dlingestion/DEMO/FINAL/EMP_DETAILS --merge-key SSN  -m 1 --class-name UTILITY_TEST.EMP_DETAILS
2 REPLIES 2

Re: Sqoop incremental import issue

@rajdip chaudhuri

You are missing the append argument.

Add --append after specifying --last-value as shown below. I have split them to multiple lines for readability.

sqoop import-Dmapreduce.job.user.classpath.first=true
--connect jdbc:oracle:thin:@//3.209.124.204:1521/eqlmpoc  --fetch-size 1000 
--username utility_test --password utility_test --table UTILITY_TEST.EMP_DETAILS 
--check-column HIREDATE 
--incremental lastmodified 
--last-value "2017-05-28 16:01:12" 
--append 
--target-dir  hdfs://nn.tcsgegdc.com:8020/data/dlingestion/DEMO/FINAL/EMP_DETAILS 
--merge-key SSN  -m 1 --class-name UTILITY_TEST.EMP_DETAILS

See this link for more details : https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

Re: Sqoop incremental import issue

New Contributor

I don't think append is require as you are using incremental in query. If this will execute successfully it will insert 3 record into table. 1 new record and 2 updated record as new row.

Don't have an account?
Coming from Hortonworks? Activate your account here