Reply
New Contributor
Posts: 1
Registered: ‎08-28-2017

Sqoop Incremental Append

My database has a timestamp column on the basis of which Iam performing my sqoop incremental import with last modified clause.

But

If I am giving last modified check column value as 11am it doesn't retrieves the records which were inserted at 11am it imports records after that.

How do I import the records processed at 11am.

I don't want to have any duplicate records or any missing records.

Champion
Posts: 777
Registered: ‎05-16-2016

Re: Sqoop Incremental Append

its is always recommended to run this as sqoop job so that you will have your last value being recorded automatically. 

 

would you consider performing 

 

--incremental append 

with 

--check-column

 which specifies the column to be examined when determining which rows to import.

 will insert all the new rows based on the last value

Highlighted
Explorer
Posts: 14
Registered: ‎03-04-2019

Re: Sqoop Incremental Append

Dear All.

I have table in sql server that column contain random unique number there is no any primary key but we want to perform incremental append or lastmodified operation using sqoop so please help me.

 

 

 

Note:-This is Critical Issue.

 

 

Thanks

HadoopHelp

Champion
Posts: 777
Registered: ‎05-16-2016

Re: Sqoop Incremental Append

You can perform lastmodified option.

Something like the below 

sqoop import \
--connect 
--username
--password 
--table 
--incremental lastmodified \
--check cloumn last_updated_date_or anything that is according to your table
--last--vaule " 2101-02-22 01:02:12"

Champion
Posts: 777
Registered: ‎05-16-2016

Re: Sqoop Incremental Append

Let me know if you need any more information. 

Explorer
Posts: 14
Registered: ‎03-04-2019

Re: Sqoop Incremental Append

Thanks!

 

but there is no any time/date column in my table?

then how can we perform last-modified operation?

 

 

 

 

Thanks

HadoopHelp

Cloudera Employee
Posts: 40
Registered: ‎01-07-2019

Re: Sqoop Incremental Append

I recall you don't need the column to be a date, but for squoop to know which records are added/changed after the point where you already got, you do need to have something incremental.

 

If you have no column that can be easily used to determine whether a row is newer or not, the only conceptual way to know whether a row is new, would be by keeping track of which values have already been loaded. This administration is very heavy something that tools like sqoop cannot do automatically.