New Contributor
Posts: 1
Registered: ‎10-04-2018

Problem with incremental append



i'm having problems with incremental append

here's what i do:


first big import:

sqoop import --connect 'jdbc:sqlserver://SERVER;databasename=DATABASE' --username 'USERNAME' --password 'PASSWORD' --query "select * from TABLE  Where \$CONDITIONS" --hive-import --target-dir=/user/hive/warehouse/TABLE --hive-table TABLE --delete-target-dir --split-by KEY


append command:

sqoop import --connect 'jdbc:sqlserver://SERVER;databasename=DATABASE' --username 'USERNAME' --password 'PASSWORD' --query "select * from TABLE  Where \$CONDITIONS" --hive-import --target-dir=/user/hive/warehouse/TABLE --hive-table TABLE --delete-target-dir --split-by KEY --incremental append --check-column KEY


Everything goes right but i don't find new rows in my hive table (same result adding --last-value parameter)


What do i do wrong?


Thank you

Cloudera Employee
Posts: 725
Registered: ‎03-23-2015

Re: Problem with incremental append

Can you try to use "--table TABLE" instead of "--query"?

Also, try to add "--verbose" to see debug messages so that you can see what Sqoop was doing.

If you are doing incremental append, you probably don't want to "--delete-target-dir", since it is --hive-import, you might not need --target-dir either.

You should have --last-value specified when doing incremental imports to start from where you left off last time.
New solutions